Indexing/robots/crawling ??? No idea :(

  • 1
  • Question
  • Updated 9 years ago
  • Answered
Hi there, last night I was reading some threads here and one was written by Peter, I think, anyways, must say I had and still have absolutely NO idea what it all meant so if someone would please explain in a simple term so someone like myself could understand it would be appreciated :)

There was talk of indexing, ok so what is indexing please? then about robots and google crawling over a site - heck.....

I have tried to find the answer but feel it's best I ask someone to explain if anyone has the time.

Thanx in advance.

INDI
Photo of INDI

INDI

  • 189 Posts
  • 2 Reply Likes
  • a lil stumped on what it all means....lol

Posted 9 years ago

  • 1
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Hello Indi,

Indexing is the process of collecting data on your web-site and categorising it in an index so that users when they do a search can find material that's categorised in the index eg: Google , Yahoo etc.

Some people don't want their sites included in the indexes. The easiest way to prevent this is to insert a small text file you call "robots.txt". This small file includes some instruction not to index the web site. Now this file must be placed in a very specific position in a web page to be seen by the robots.

Robot is a term for a program written by Google or whoever to continuously scan web pages for information . If it sees a "robots.txt" and if the small file contains instructions not to include this site in the "index" it follows this instruction.

Synthasite hasn't given access for users to place a robots.txt file in the mentioned specific area of their site. So Synthsite users cannot use robots.txt to prevent indexing.

Summary of terms

"Indexing": the process of collecting the information on the website for the purpose of palcing the site in an index eg Google, Yahoo etc.

"robot": the program that "crawls" over and through web-sites looking for the information and instructions.

Why do people not want their sites or part of their sites not included? Privacy, redundancy many things.

The procedures do not give your undiscoverable web-sites. That's a whole different issue.
Photo of INDI

INDI

  • 189 Posts
  • 2 Reply Likes
Thank you so much for your response Peter, and best of all I now understand what it was you were talking about :) Your knowledge and sharing of info is very much appreciated.

I note you said that Synthasite hasn't given access for users to place a robot.txt file but would this be different if a person has their own domain or does it still have no access.

I am still currently working on my site as I have so much to input but I'd like to get it as close to 100% perfect when comes time to publish it so the more I learn now whilst still building means the more things I can implement before publishing rather than after.

This bit is probably a bit off topic to my original post sorry, but can I get my own domain whilst still working on my site or do I have to wait until it's time to publish it?

Thank you Peter once again for each time you have answered my posts with detailed yet understandable answers for me and also to everyone else here who is only too happy to help others out.

INDI
Photo of Peter

Peter

  • 2569 Posts
  • 113 Reply Likes
Hi Indi,

The inability to place a robots.txt file is a function of synthasite pages. It doesn't make any difference if it's a domain or a sub-domain. No access currently. It's under consideration indi.

You should be able to get your domain prior to publishing. Send an email to support@synthasite.com.
Photo of Marije

Marije, Official Rep

  • 4636 Posts
  • 237 Reply Likes
Hi Indi,

Peter has given some fantastic advice (thanks Peter!).

You can definitely purchase your domain before your site is finished. Once it's done, the "Publish my Site" button changes to "Update my Site" so that you can update your website whenever you make changes.

If you want to purchase your domain but keep your site private until it's ready, you can always password protect it. This link has more info: http://getsatisfaction.com/SynthaSite...

Let us know if you need any other help.
Photo of INDI

INDI

  • 189 Posts
  • 2 Reply Likes
Thank you both for your replies :)

I guess the reason I was wanting to keep my site off a search engine is because my site is mostly directly at locals and Aussie's (bit hard to explain just now) and pretty much would only be found by persons I gave the domain addy to etc.

I'll have to get some money into my Paypal account to get my domain name, once i've secured that I can finish printing some small cards with the URL on it and know that it's mine :)

Thanks again.

Indi