ROBOTS.TXT FILE ACCESS

  • 1
  • Question
  • Updated 7 years ago
  • Answered
Could you please tell how to get access to robots.txt file

I would like to
add the following two lines of text to the top of your robots.txt file:
User-agent: Mediapartners-Google
Disallow:

to allow Google crawler

Regards
Venkta Lakkoju
Photo of venkata

venkata

  • 1 Post
  • 0 Reply Likes

Posted 7 years ago

  • 1
Photo of Marije

Marije, Official Rep

  • 4636 Posts
  • 237 Reply Likes
Hello Venkata,

A robots.txt file is a file that informs search engines which files/directories of the site you do not want indexed. If you do not add a robots.txt file then your entire site will be indexed.

See the Google help page for more information on this topic. Are there any parts of your site that you do not want indexed? If not, you do not need to add a robots.txt file to your site.

If you get a message saying that your URL is blocked because of a robots.txt file, please ignore this. Google has simply tried to index the preview URL of one of your pages. A preview URL is what you see in your browser window when you open your site in the Yola Site Builder and then preview it. Yola prevents Google from crawling these URLs as Google should not index unpublished sites.

This message simply serves to indicate that everything is working correctly. This will not interfere with your search engine ranking or Google Webmaster's ability to return data about your site. As long as your site is published and you have submitted the correct URL, Google will be able to crawl your site.

If you get a "Page not Found" or 404 error this probably means that there was a problem with your DNS server (the server of your domain name registrar) or that you had made a DNS change around the time that Google tried to index your site. Google makes an attempt approximately every 5 days, so you simply need to wait for the Google spiders to come around again and everything should be fine.