How to prevent Google and others from Indexing your site (summary)

  • 2
  • Idea
  • Updated 10 years ago
Cosmic Sensorium replied 15 days ago
hey guys, an option you might consider is using noindex meta attributes. it's not really equivalent to a robots.txt file, but the tag can tell search engines not to index pages in your site.

To prevent all robots from indexing a page on your site, drop an html widget on your page and place the following meta tag in it (without the <!--- and --->, that's just there so that the code displays):

<!---
<meta name="robots" content="noindex">
--->

If you only want to keep Google from indexing your page, place this code instead:
<!---
<meta name="googlebot" content="noindex">
--->

here's some more info:
http://www.google.com/support/webmast...

and here's a comparison between using noindex, nofollow and the robots.txt file:
http://www.seobook.com/robots-txt-vs-...

hope this helps!

I’m at http://CosmicSensorium.com

Inappropriate?
Cosmic Sensorium replied 2 hours ago
Hi,

This is the link I had posted:

http://www.google.com/support/webmast...

<!---
<meta name="robots" content="noindex">
--->

will tell robots not to index the particular pages you put this code in (remember to just copy what's between the <!--- and --->)

So far, the only way I know of to add this is dropping an html widget and pasting the code on each page you want not to be indexed... The thing is it does not end up in the head section of your site (where meta tags are supposed to be). Nonetheless, dropping this particular meta tag outside of the head section has worked for me. The pages I put it on are not being indexed by google according to webmaster tools.

I also tested this by dropping the code on pages that had already been indexed. They were removed from the index the next time my site was crawled by google. Taking the code off allows for your page to be indexed on subsequent crawls.

Hope this helps!

D

1 Comment Add a comment This answers the question

Cosmic Sensorium commented 2 hours ago
One more thing - the reason why I chose to do this instead of password protection is the following. I changed my site around; deleted some sections, changed the names of some pages, etc. Some of the pages I intended to delete had incoming links from other sites. So I decided to leave the pages there and put in redirects to the new corresponding section. When doing this, Webmaster Tools gave me a warning about the redirects.

I wanted the pages to be there, and for them to be accessible by users following links to them - I just did not want google to index those pages. Password protection would not have been useful for users following links to those pages, as ultimately they would not have been able to reach the content they were looking for.
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
  • thankful

Posted 10 years ago

  • 2
Photo of Cosmic Sensorium

Cosmic Sensorium

  • 102 Posts
  • 5 Reply Likes
Nice summary! I was trying to find that original reply earlier on...
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Thank you for your work on this Diogenes.

I was trying to get your post into a searchable headline as the question comes up frequently and as you have said "trying to find that original" is a time taker. I think because it's caught up in the middle of other things.

My experience with this is similar to yours but now I can't verify my site. (Which doesn't worry me in the least).Coincidence or cause I don't know.

I'm running it again as a trial with a new site so maybe I'll have another reference point in a few days to weeks. Also have asked Synthasite if they can trial it then we should have the definitive answer, or near definitive as I see Google has mentioned that there's a possibility of it not working occasionally.
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Hi Diogenes and Peter

I appreciate your research on this, as a number of people have this question.

We are trying the "no index" metatag on a site that is currently indexed by Google. If it disappears from the search results the next time the site is crawled we will have a pretty good indication that this method works, and we will let you know.
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Thanks Monique.