help with my website

  • 1
  • Question
  • Updated 10 years ago
  • Answered
How do I grant access to your crawler?
If you would like to grant our crawler access to your pages, you can do so without granting permission to any other bots. Simply add the following two lines of text to the top of your robots.txt file:

User-agent: Mediapartners-Google
Disallow:

This change will allow our crawler to index the content of your site and provide you with the most relevant Google ads for that content.

Was this article:

is there any way you could help me with this problem
Photo of jeremy

jeremy

  • 6 Posts
  • 0 Reply Likes
  • indifferent

Posted 10 years ago

  • 1
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Hi Jeremy,

It seems this is the day for the same or similar questions as yours. You are the third today.

The robot.txt file is a very simple text file but users cant' access it in the root directory. Monique at Synathsite has suggested that if there's a growing need this it could be reviewed in the future, but for now we can't block the Google crawler and prevent Google indexing Synthasites.
Photo of Emmy

Emmy

  • 5892 Posts
  • 299 Reply Likes
Hello Jeremy,
Yes, Peter is right, it is not possible at this time to edit the robots.txt file.

Do you have a page that is password protected? Have you gone through verifying your site with Google Webmaster using the META option? Please let me know and I will help further.

Thanks,
Emmy
Photo of jeremy

jeremy

  • 6 Posts
  • 0 Reply Likes
Yes I do have a password protected page. should i take it off to allow google to access my page. and what do you mean by verifying my site with google webmaser usining the META option.
Photo of Emmy

Emmy

  • 5892 Posts
  • 299 Reply Likes
Hello Jeremy,
According to Google AdSense it is necessary for you to create authentication parameters for a password protected page to be accessed, which can be done with Google Webmaster. Please see this page of Google AdSense help area: How do I allow your crawler access to my login protected page.

We have specific instructions for this (creating authentication parameters) that you can follow which will help you verify your site with Google Webmaster using the META option. Please follow trough this tutorial Google Webmaster Tutorial to verify your site.

This should clear up the problem you are having with AdSense being able to access you password protected page. If it doesn't, please let us know so we can look into this further and get it figured out.

KInd Regards,
Emmy