changing robots.txt

  • 1
  • Question
  • Updated 9 years ago
  • Answered
Russ CarewYola Expert
I was trying to upload and update the robote.txt. for Attracta sitemap. so I can get a better score, and get more people to my site. The directions for uploading the robots. txt suck. and my yola site is not much help either. what can i do next to change the robots.txt. and get google and yahoo. to update my listing.
Photo of RussC


  • 2 Posts
  • 0 Reply Likes

Posted 9 years ago

  • 1
Photo of Brianna


  • 2045 Posts
  • 92 Reply Likes
Hi Russ,

A robots.txt file is a file that informs search engines which files/directories of the site you do not want indexed. If you do not add a robots.txt file then your entire site will be indexed.

See the Google Help page for more information on this topic. Are there any parts of your site that you do not want indexed? If not, you do not need to add a robots.txt file to your site.

If you get a message saying that your URL is blocked because of a robots.txt file, please ignore this. Google has simply tried to index the preview URL of one of your pages. A preview URL is what you see in your browser window when you open your site in the Yola Site Builder and then preview it. Yola prevents Google from crawling these URLs as Google should not index unpublished sites.

This message simply serves to indicate that everything is working correctly. This will not interfere with your search engine ranking or Google Webmaster's ability to return data about your site. As long as your site is published and you have submitted the correct URL, Google will be able to crawl your site.

If you get a "Page not Found" or 404 error this probably means that there was a problem with your DNS server (the server of your domain name registrar) or that you had made a DNS change around the time that Google tried to index your site. Google makes an attempt approximately every 5 days, so you simply need to wait for the Google spiders to come around again and everything should be fine.

Certainly let us know if you need further clarification.