If you need support from Yola please contact us via firstname.lastname@example.org or make use of our live chat service! We are working on a new approach to community-based support and during this transition period we are not adding any new threads to this forum. Many thanks to everyone who has participated over the years and we look forward to unveiling a new peer-to-peer support system soon!
A robots.txt file is a file that informs search engines which files/directories of the site you do not want indexed. If you do not add a robots.txt file then your entire site will be indexed.
See the Google help page for more information on this topic. Are there any parts of your site that you do not want indexed? If not, you do not need to add a robots.txt file to your site.
If you get a message saying that your URL is blocked because of a robots.txt file, please ignore this. Google has simply tried to index the preview URL of one of your pages. A preview URL is what you see in your browser window when you open your site in the Yola Site Builder and then preview it. Yola prevents Google from crawling these URLs as Google should not index unpublished sites.
This message simply serves to indicate that everything is working correctly. This will not interfere with your search engine ranking or Google Webmaster's ability to return data about your site. As long as your site is published and you have submitted the correct URL, Google will be able to crawl your site.
If you get a "Page not Found" or 404 error this probably means that there was a problem with your DNS server (the server of your domain name registrar) or that you had made a DNS change around the time that Google tried to index your site. Google makes an attempt approximately every 5 days, so you simply need to wait for the Google spiders to come around again and everything should be fine.