Prevent Google from crawling on pages other than homepage

  • 1
  • Question
  • Updated 9 years ago
  • Answered
How to prevent Google from crawling through other pages of my site other than the homepage, because I only set up password to protect my homepage and I don't want to put password on other pages as it'll be very troublesome to need to enter password every time I enter a page. Hopefully someone is able to help me with this. Thank you very much.
Photo of Bruce

Bruce

  • 4 Posts
  • 0 Reply Likes

Posted 9 years ago

  • 1
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Bruce,

If you put a password protection on your homepage I know that Google won't include this page in the index.

I don't know if this prevents subsequent non-protected pages from being indexed but I suspect that it doesn't.

Currently I am exploring other options for this and I hope to have answers within a few days. As you're probably aware at this time we cannot use "robots.txt" as a solution with Synthasite.

Diogenes posted a tip sometime ago about selective and universal blocking of pages and others have doubted its effectiveness. So we want to try this and come up with a hopefully definitive answer.

Synathsite have put the issue of the "robots.txt" onto their development plan and this could appear either soon or later.
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Hi Bruce

To add to Peter's excellent and thorough response:

If you password protect all the pages on your site, people will only need to log in once from any of the protected pages to gain access to all the other protected pages. (We had a bug where you were asked to log in on every page a few weeks ago, but this has been fixed now)

You can't upload a robots.txt file because this requires you to have access to the root directory of your site - which you don't at this time. In future we hope to find a way to allow users to edit their robots.txt file.

In the meantime my best suggestion is that you password protect all the pages on your site.
Photo of Cosmic Sensorium

Cosmic Sensorium

  • 102 Posts
  • 5 Reply Likes
Hi,

This is the link I had posted:

http://www.google.com/support/webmast...

<!---
<meta name="robots" content="noindex">
--->

will tell robots not to index the particular pages you put this code in (remember to just copy what's between the <!--- and --->)

So far, the only way I know of to add this is dropping an html widget and pasting the code on each page you want not to be indexed... The thing is it does not end up in the head section of your site (where meta tags are supposed to be). Nonetheless, dropping this particular meta tag outside of the head section has worked for me. The pages I put it on are not being indexed by google according to webmaster tools.

I also tested this by dropping the code on pages that had already been indexed. They were removed from the index the next time my site was crawled by google. Taking the code off allows for your page to be indexed on subsequent crawls.

Hope this helps!

D
Photo of Cosmic Sensorium

Cosmic Sensorium

  • 102 Posts
  • 5 Reply Likes
One more thing - the reason why I chose to do this instead of password protection is the following. I changed my site around; deleted some sections, changed the names of some pages, etc. Some of the pages I intended to delete had incoming links from other sites. So I decided to leave the pages there and put in redirects to the new corresponding section. When doing this, Webmaster Tools gave me a warning about the redirects.

I wanted the pages to be there, and for them to be accessible by users following links to them - I just did not want google to index those pages. Password protection would not have been useful for users following links to those pages, as ultimately they would not have been able to reach the content they were looking for.
Photo of Bruce

Bruce

  • 4 Posts
  • 0 Reply Likes
Thank you so much guys for your time to reply to my question. Hopefully I'll be able to help back next time in the future. Thanks.