URLs restricted by robots.txt ... ?

  • 1
  • Question
  • Updated 11 years ago
I'm getting the "URLs restricted by robots.txt" message and just wanted to make sure that this is ok/normal when google can't read these. In my case, what google cant see is a few pdf files, that I saved from the internet. Is this normal? Is there anyway I can get rid of the errors as I like to have a clean bill of health, if you see what I mean?
Photo of Will

Will

  • 252 Posts
  • 4 Reply Likes
  • happy that things are going so well

Posted 11 years ago

  • 1
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Official Response
Hi Will

All Yola sites have a robots.txt file associated with them to prevent Google from crawling parts of your site that should not be indexed. If you want to see your site's robots.txt file, type your domain name into your browser, followed by /robots.txt. You will then see the URLs that are disallowed. One of these URLs is /resources. However, Google will be able to index the file once you embed it on your page.

However, there is some discussion about how Google handles the indexing of PDF files. Here is a useful forum discussion on the subject: http://www.webmasterworld.com/forum10...

You should not be concerned about this message as it does not indicate a problem with your site.
Photo of RSPublishing

RSPublishing

  • 298 Posts
  • 9 Reply Likes
i have exactly the same problem. my adsense showing several blocked url/robots.txt. also, my site displaying irrelevant ads!!!!! quick question, does the ads display differently in different countries or does the same ads i see displayed, gets seen by everyone visiting my website? How sure are you (Yola), that these restricted/blocked robots are nothing to worry about? does this in any way affect our adsense ads/rankings/etc? Last question, if this is a subdomain, can i still rank well in google? Please check out my website and let me know about any problems i am missing. Its an informative medical website. (Applied for my HON seal of approval from Health on the net code of conduct).
http://www.mesotheliomacancer.yolasit...
Photo of Ruth

Ruth

  • 2819 Posts
  • 135 Reply Likes
Hi Rynaldo,

As you add more content to your site and your site is crawled with more frequency your ads should become more relevant because Google will learn what your site is about. This will take time, and necessitate you updating your content on a regular basis.

People have had very good success with Google using a free subdomain name, but you may wish to purchase a custom domain name as it can look more professional and be easier for your visitors to remember.

Without seeing the exact message that you received about the blocked url/robot.txt there is no way for us to be 100% sure that this is your situation, but I am very confident that this is the same case as outlined above.

Good luck with your site. Please let us know if there is anything else we can help you with
Photo of Will

Will

  • 252 Posts
  • 4 Reply Likes
That's a very clear and concise answer. Thanks so much.