Proper robots.txt disallow format

fizzer

Senior Member

Join Date: Oct 2011
Posts: 104

04-08-2014, 03:56 AM

Hello, i found 3 ways to disallow some robots from indexing a file, please help me to point out which are incorrect and which you think is best?

#1

Code:

User-Agent: *
Disallow: 


User-Agent: Googlebot
Disallow: /index-ads.html


User-Agent: Googlebot-Mobile
Disallow: /index-ads.html


User-Agent: Googlebot-Image
Disallow: /index-ads.html


User-Agent: Mediapartners-Google
Disallow: /index-ads.html


User-Agent: Adsbot-Google
Disallow: /index-ads.html


User-Agent: Slurp
Disallow: /index-ads.html


User-Agent: msnbot
Disallow: /index-ads.html


User-Agent: msnbot-media
Disallow: /index-ads.html


Sitemap: http://mysite.com/sitemap.php

Code:

User-Agent: Googlebot
User-Agent: Yahoobot
User-Agent: Microsoftbot
Disallow: /index-ads.html


User-agent: *
Disallow: /downloads/
Allow: /

Code:

User-Agent: Googlebot Googlebot-Mobile Googlebot-Image Mediapartners-Google
Disallow: /index-ads.html
Disallow: /downloads/
Allow: /

I like a third one, but im not sure if its valid.

Thank you

Last edited by fizzer; 04-08-2014, 03:58 AM.

- Dedicated and virtual servers payable using ******currencies
- Hosting Forum

Tags: robots

Previous template Next

1 of 2 < >

FreeHostForum Rules and Guidelines

Webmaster forum - Web Hosting Forum,Domain Name Forum, Web Design Forum, Travel Forum,World Forum, VPS Forum, Reseller Hosting Forum, Free Hosting Forum

Signature

Board-wide Policies:

Do not post links (ads) in posts or threads in non advertising forums.

Forum Rules
Posts are to be made in the relevant forum. Users are asked to read the forum descriptions before posting.

Members should post in a way that is respectful of other users. Flaming or abusing users in any way will not be tolerated and will lead to a warning or will be banned.

Members are asked to respect the copyright of other users, sites, media, etc.

Spam is not tolerated here in most circumstances. Users posting spam will be banned. The words and links will be censored.

The moderating, support and other teams reserve the right to edit or remove any post at any time. The determination of what is construed as indecent, vulgar, spam, etc. as noted in these points is up to Team Members and not users.

Any text links or images contain popups will be removed or changed.

Signatures
Signatures may contain up to four lines

Text in signatures is subject to the same conditions as posts with respect decency, warez, emoticons, etc.

Font sizes above 3 are not allowed

Links are permitted in signatures. Such links may be made to non-Freehostforum material, commercial ventures, etc. Links are included within the text and image limits above. Links to offensive sites may be subject to removal.

You are allowed ONLY ONE picture(banner) upto 120 pixels in width and 60 pixels in height with a maximum 30kB filesize.

In combination with a banner/picture you can have ONLY ONE LINE text link.

Advertising
Webmaster related advertising is allowed in Webmaster Marketplace section only. Free of charge.

Shopping related (tangible goods) advertising is allowed in Buy Sell Trade section only. Free of charge.

No advertising allowed except paid stickies in other sections.

Please make sure that your post is relevant.

More to come soon....

2 of 2 < >

Advertise at FreeHostForum

We offer competitive rates and a many kinds of advertising opportunities for both small and large scale campaigns.More and more webmasters find advertising at FreeHostForum.com is a useful way to promote their sites and services. That is why we now have many long-term advertisers.

At here, we also want to thank you all for your support.

For more details:
http://www.freehostforum.com/threads...eHostForum-com

More ad spots:
http://www.freehostforum.com/forums/...-FreeHostForum

See less

AD MANAGEMENT

BEHOSTED

Proper robots.txt disallow format