PDA

View Full Version : How to create robot.txt File ?



rajeevmehtaa
08-11-2011, 09:16 AM
How to create robot.txt File ?

jcworlditsme
08-11-2011, 13:36 PM
This is relatively easily, you just need to open Notepad and save a text file as robots.txt, for information on what to put in these files, search Google for "robots.txt".

tylerlongtw
08-11-2011, 14:18 PM
Hello jcworlditsme,

I like your amazing information.
Thanks

ramsee
04-06-2012, 03:10 AM
Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from Tucktail.com (http://www.tucktail.com/)

A robots.txt is a simple text file that can be created with Notepad. each robot.txt file should contain two parts
1. User agent: * means all search bots should use the instruction to crawl the website. unless your site is a complex one you need not seperate instructions for seperate spiders.
2.disallow : contains the files which should be avoided from crawling.
Eg.
User-agent: *
Disallow: /wp-

diyakapoor
04-06-2012, 08:54 AM
Robots.txt file is created to stop crawling access on some of the pages. You can easily find code for this in Google. Just copy and paste the code and give the page/folder urls you want to stop access and save the file as robots.txt format and upload at root of your web server