help me

  • 1
  • Question
  • Updated 9 years ago
  • Answered
what is a robots.txt file, how do I know if it allows google to crawl it, and if it doesn't, how do I change it?
Photo of brian


  • 1 Post
  • 0 Reply Likes
  • angry

Posted 9 years ago

  • 1
Photo of Ruth


  • 2819 Posts
  • 135 Reply Likes
Hi Brian,

A robots.txt file is a file that informs search engines which files/directories of the site you DO NOT want indexed. If you do not add a robots.txt file then your entire site will be indexed.

See: for more information on this topic.

Are there any parts of your site that you do not want indexed? If not, you do not need to add a robots.txt file to your site.

If you get a message saying that your URL is blocked because of a robots.txt file, please ignore this. Google has simply tried to index the preview URL of one of your pages. A preview URL is what you see in your browser window when you open your site in the Yola sitebuilder and then preview it. Yola prevents Google from crawling these URLs as Google should not index unpublished sites.

This message simply serves to indicate that everything is working correctly. This will not interfere with your search engine ranking or Google Webmaster's ability to return data about your site.

I hope this information is helpful to you. Please let us know if we can help you with anything else.