Google Webmasters Sitemap?

  • 1
  • Question
  • Updated 8 years ago
I'm trying to help my brother with his website - its in the very very very early stages but in google webmasters we read this on the sitemap page. Is this ok? i was told that the / after dissallow prevents pages being crawled? and surely he wants them crawled?
User-agent: *
Sitemap: http://www.xxxxxxxxxx.yolasite.com/si...
Disallow: /classes/
Disallow: /definitions/
Disallow: /templates/

Am i missing something? or is this how its supposed to be.
Photo of Grant Hards

Grant Hards

  • 9 Posts
  • 0 Reply Likes
  • unsure

Posted 8 years ago

  • 1
Photo of Laura Thomas

Laura Thomas, Social Media Coordinator

  • 4536 Posts
  • 211 Reply Likes
Hi Grant,

Is your brothers website published to the web yet? If so, can you please reply with the web address so we can take a closer look?
Photo of Grant Hards

Grant Hards

  • 9 Posts
  • 0 Reply Likes
www.gbcleaningservices.co.uk i think thats right
Photo of SilverX

SilverX

  • 492 Posts
  • 40 Reply Likes
Robots.txt file simply prevents bots from being crawling the site's specific parts as long as that bot obeys to robots.txt protocol.

Disallow: /templates/

above line means that "do not *crawl* (not index - if you want to get that page crawled but not included in search results, this is not the way. For an example, if you add a link to my site in a page that restricted by robots.txt, that link is completely worthless) all links followed by /templates.

/templates/x.php
/templates/x/y.php

will not be crawled.

However if you want to restrict only /templates/x.php,

Disallow: /templates/$

restricts only /templates/x.php

in your case, you can safetly ignore that message.
If you checked my site's robots.txt, you'll see many more restricted areas.

/templates, /classes, etc are Yola system files that shouldn't be crawled.
Photo of Sanja

Sanja, Official Rep

  • 10698 Posts
  • 495 Reply Likes
Thanks Silverx for this information.
Photo of Sanja

Sanja, Official Rep

  • 10698 Posts
  • 495 Reply Likes
Hello Grant

I'm going to also alert our SEO team of this thread to see if they have any additional information they can share. Please allow them 48 hours to respond. Thanks!
Photo of Emmy

Emmy

  • 5892 Posts
  • 299 Reply Likes
Hello Grant,

Yes, there is a robots.txt file that is set to disallow search engine spiders from crawling those directories of your site. These are not content areas, such a the pages of your site, and the data stored in these directories is not pertinent to the search engines. This robots.txt file that disallows these directories is in place for all Yola websites and will not impeded your site's searchability or indexing. I hope that helps!

Emmy
Photo of Grant Hards

Grant Hards

  • 9 Posts
  • 0 Reply Likes
Thank you all :)