Webmaster Forum  

Go Back   Webmaster Forum > Web Hosting Forum - Web Hosting - Web Host - Hosting - Managed Hosting - Shared Hosting > Site & Server Administration
User Name
Password
Register FAQ Members List Calendar Transactions Store Search Today's Posts Mark Forums Read


robot.txt

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 04-30-2008, 01:14
puffin
Senior Member

puffin's Avatar

Join Date: Nov 2006
Posts: 778
Trader Rating: (0)
Points: 23 (Donate)
5 F$/Referral Refer Friends
puffin is on a distinguished road
Default robot.txt

Well i want to do some testing in my forum so i made a clone of it in another domain i want to stop search engines spidering it and index. How to do it with a robot txt file ?
puffin is offline
Reply With Quote
  #2  
Old 05-17-2008, 20:58
Raider
Junior Member


Join Date: May 2008
Posts: 1
Trader Rating: (0)
Points: 0 (Donate)
5 F$/Referral Refer Friends
Raider is on a distinguished road
Default

User-agent: *
Disallow: /
Raider is offline
Reply With Quote
  #3  
Old 05-18-2008, 21:56
mafrussian
Junior Member


Join Date: May 2008
Posts: 27
Trader Rating: (0)
Points: 0 (Donate)
5 F$/Referral Refer Friends
mafrussian is on a distinguished road
Default

Thanks raider, I've been looking for that everywhere. I am so glad that someone was finally capable of answering this question
mafrussian is offline
Reply With Quote
  #4  
Old 06-16-2008, 15:53
puffin
Senior Member

puffin's Avatar

Join Date: Nov 2006
Posts: 778
Trader Rating: (0)
Points: 23 (Donate)
5 F$/Referral Refer Friends
puffin is on a distinguished road
Default

Thanks for the answer. Didn't know it was that easy. How you would do it you want to stop a particular search engine ? I'm sure is something pretty easy though I have no idea about it.
puffin is offline
Reply With Quote
  #5  
Old 06-24-2008, 09:41
Maurizio1230
Senior Member


Join Date: Sep 2007
Posts: 170
Trader Rating: (0)
Points: 0 (Donate)
5 F$/Referral Refer Friends
Maurizio1230 is on a distinguished road
Default

Can I use this robot.txt file in custom script too?
I see this file in joomla dir too but I don't know how I can use this in general script.
Maurizio1230 is offline
Reply With Quote
  #6  
Old 06-29-2008, 14:34
puffin
Senior Member

puffin's Avatar

Join Date: Nov 2006
Posts: 778
Trader Rating: (0)
Points: 23 (Donate)
5 F$/Referral Refer Friends
puffin is on a distinguished road
Default

What do you mean by a custom script file ? If you are going to build custom script and you want to stop search engines spidering pages you can use a robot.txt file. its same as we use in normal websites. Heres general commands of robot.txt file

http://www.hasitha.info/how-to-use-...gines-spidering
puffin is offline
Reply With Quote
  #7  
Old 07-02-2008, 03:42
mrinal
Member


Join Date: Jun 2008
Posts: 56
Trader Rating: (0)
Points: 1 (Donate)
5 F$/Referral Refer Friends
mrinal is on a distinguished road
Default

usually i place a clone of a site in a subfolder and use robots.txt to avoid spidering these and with any custom scripts and general CMS like joomla and pligg u can have robots.txt of own.
mrinal is offline
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
robot.txt puffin Site & Server Administration 2 01-05-2008 06:50

Resources : | Advertise at FHF | itextLink.com| Reseller Hosting| TextDot| iNamePros| Any Webmaster| Web Host| Dep3|


All times are GMT -4. The time now is 00:27.


Powered by: vBulletin
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.