robots.txt

[seomastermack]

How does one go about creating a robots.txt file and placing it in a web site?

[Nile Hadwards]

robots.txt basically tells search engine spiders not to index certain areas of your site. You can protect private areas of sites from being visible to search engines, and thus to everyone.

[John23]

Re:

As of me a robot.txt is a txt file that contains instructions in a specific format. That often used by robots of search engines to categories and archive web sites. If this robot.txt file doesn't exist then it means owner doesn't want to provide specific instruction.

[ramsee]

informative!!!

[qualityweb]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do

[john515]

Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not.

[testandtag365]

Robots.txt helps to know to the search engine to crawl the page or not. When you code in any page then there is required to know that this page is required to crawl or not then any search engine indexing of that otherwise if mention then not crawling of that.

[saina]

Robot.txt files are text files that allow and disallow the search engine to crawl specific page.