how and where do I add the "robots.txt" file

  • 3
  • Question
  • Updated 8 years ago
  • Answered
Photo of Guido Graphics

Guido Graphics

  • 558 Posts
  • 41 Reply Likes
  • frustrated

Posted 10 years ago

  • 3
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
A robots.txt file is a file with information that informs search engines which files/directories of the site to index. If you do not add a file like this your entire site will be indexed.

See: http://www.google.com/support/webmast.... Are there any parts of your site that you do not want indexed? If not, you do not need to add a robots.txt file to your site.

If you do decide to add one, you would need to upload it to your file manager.
Photo of Donnie

Donnie

  • 2 Posts
  • 1 Reply Like
I actually uploaded a robots.txt file to my file manager on 3 sites occording to these and google's instructions and google still says my url's are blocked.
Photo of Ed-a-Torials @ Honey Bear Playhomes

Ed-a-Torials @ Honey Bear Playhomes, Champion

  • 2582 Posts
  • 279 Reply Likes
Hi Donny just a quick question for you...

1) Are all the pages you have created with Synthasite those which you wish to be indexed and visible to others?

Y) If yes... then don't include a robot.txt file in your file manager and if it's there delete it. (reason being that if it's there and improperly done it can hinder rather than help) If it is not there then all your pages will be indexed by the crawlers.

N) If no... then make sure that it contains the proper additional rules as described in the (Google Webmaster Tools) then (Dashboard) then (Tools) then (Generate robots.txt) page (I can't offer a direct link because Google requires that you sign in and if I gave the link that I see it would be for my site and not yours but here is the dashboard link...

https://www.google.com/webmasters/too...

Webmaster Help Center Checking robots.txt

http://www.google.com/support/webmast...
Photo of Donnie

Donnie

  • 2 Posts
  • 1 Reply Like
Ed: The answer is yes for my current sites, but I would like to learn how to do it correctly for future sites. The issue doesn't seem to be that difficult. Google gives you 2 lines to put in the file and synthasite says put it in the file manager. Seems pretty simple, but the google crawler still returns a blocked URL error. Thank you though.
Photo of Guido Graphics

Guido Graphics

  • 558 Posts
  • 41 Reply Likes
Step away, from the robot!
Photo of http://www.lifeingeneral.synthasite.com

http://www.lifeingeneral.synthasite.com

  • 271 Posts
  • 2 Reply Likes
I've just had a look at your site, Guido. It's awesome. Very slick....
Photo of Guido Graphics

Guido Graphics

  • 558 Posts
  • 41 Reply Likes
Thanks, appreciate that. If you have a minute, please visit http://forsite.synthasite.com and vote for #4 :) Peace, bra. Guido
Photo of Boomer

Boomer, Champion

  • 2055 Posts
  • 84 Reply Likes
Hey GG,
I will be taking a new screenshot of your site soon - I haven't forgotten.
I'm busy with a new website page - show you soon.
Your site looks #@% cool btw... It's a lowslider now.
Photo of Guido Graphics

Guido Graphics

  • 558 Posts
  • 41 Reply Likes
Grazie, bommerdude! By the way... I'm up to 405 visits from forsite! Ubuntu!
Photo of sean kelly

sean kelly

  • 13 Posts
  • 0 Reply Likes
Hi,i have a similar problem on yola.Googles crawler is reading the SITE BUILDER not the actual url (which is correct)the site is fine,my ads are fine earnings are down but i hear everyones is.I dont have pages i dont want crawled so what should i do to remedy this? cheers
p.s i have a test page that isnt in the menu-is this causing the problem?
Photo of Monique

Monique, VP of Customer Support

  • 6294 Posts
  • 446 Reply Likes
Can you post the full error message you get from Google? It could be that Google attempts to read the sitebuilder page but gets blocked by a robots.txt file. If this is the case then this is the correct behavior and you can simply ignore the message. Google should be able to read the code on the live site and it should not interfere with your Adsense earnings.