URLs restricted by robots.txt ... ?

  • 1
  • Question
  • Updated 10 years ago
I'm getting the "URLs restricted by robots.txt" message and just wanted to make sure that this is ok/normal when google can't read these. In my case, what google cant see is a few pdf files, that I saved from the internet. Is this normal? Is there anyway I can get rid of the errors as I like to have a clean bill of health, if you see what I mean?
Photo of Will

Will

  • 252 Posts
  • 4 Reply Likes
  • happy that things are going so well

Posted 10 years ago

  • 1
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Official Response
Hi Will

All Yola sites have a robots.txt file associated with them to prevent Google from crawling parts of your site that should not be indexed. If you want to see your site's robots.txt file, type your domain name into your browser, followed by /robots.txt. You will then see the URLs that are disallowed. One of these URLs is /resources. However, Google will be able to index the file once you embed it on your page.

However, there is some discussion about how Google handles the indexing of PDF files. Here is a useful forum discussion on the subject: http://www.webmasterworld.com/forum10...

You should not be concerned about this message as it does not indicate a problem with your site.