|
|
|
#1
12-30-2007, 13:49
|
||
|
robot.txt
How to stop search engines crawling pages like contact pages or login pages using robots.txt. ? what should we write there for this ?
__________________
|
|
|
|
|
|
#2
12-31-2007, 06:10
|
||
|
Disallow: /<subdirectory/file name>
for best details look at google's robots.txt http://google.com/robots.txt |
|
|
|
|
|
|
#3
01-05-2008, 07:50
|
||
|
You could use the following code
User-agent: * Disallow: /members You could replace * with the robot name like googlebot (or ) to block all search engines you could have the * mark. Disallow field is to block a particular file or a directory. To more about robots.txt visit the following url. http://www.robotstxt.org/ |
|
|
|
|
|
 |
| Thread Tools | Search this Thread |
| Display Modes | |
|
|
Linear Mode
