what the heck is a robot.txt file?

  • 1
  • Problem
  • Updated 8 years ago
  • Solved
Hello. I have just activated google adsense for 2 of my sites, and immediately rec'd a "blocked URL" message, due to two missing lines from my robot.txt file. Can you please tell me how/where to add the two lines?
Photo of Gwen Norris

Gwen Norris

  • 3 Posts
  • 0 Reply Likes
  • not worried . . not yet!

Posted 8 years ago

  • 1
Photo of diane


  • 1238 Posts
  • 121 Reply Likes
Hi, Gwennie80206,

The URL in that message from Google is your preview URL (the URL of your unpublished site when you preview it) A robots.txt file blocks Google from indexing parts of your site and this would prevent them from placing ads on your site. Our unpublished sites have robots.txt files attached to them. This is in order to prevent Google from indexing your site before you publish it. As soon as your site is published Google can crawl your site and place ads.

This is what happens:

1. You paste AdSense code on your site.
2. You preview your site. Your previewed page acts like a web page with AdSense code, so Google attempts to index it, but we block Google from indexing your preview URL with a robots.txt file.

That is why you get this message. Please ignore it. It does not mean that you need to do anything. It has no impact on Google's ability to index your published website and it will not interfere with Google's ability to place advertising code on your site or with your advertising revenue. Once your site is published and the ads are displaying on your site, you can safely ignore that message. Over time, as you update your content and develop your site, the Google Ads will become increasingly relevant.