|
|
|
|
|
#1
12-30-2007, 12:49
|
||
|
robot.txt
How to stop search engines crawling pages like contact pages or login pages using robots.txt. ? what should we write there for this ?
__________________
|
|
|
|
|
|
#2
12-31-2007, 05:10
|
||
|
Disallow: /<subdirectory/file name>
for best details look at google's robots.txt http://google.com/robots.txt |
|
|
|
|
|
|
#3
01-05-2008, 06:50
|
||
|
You could use the following code
User-agent: * Disallow: /members You could replace * with the robot name like googlebot (or ) to block all search engines you could have the * mark. Disallow field is to block a particular file or a directory. To more about robots.txt visit the following url. http://www.robotstxt.org/ |
|
|
|
|
|
 |
| Thread Tools | Search this Thread |
| Display Modes | |
|
|
Hybrid Mode
