GOOGLE

Collapse

XHOST

Collapse

What is Robots.txt?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.
    MilesWeb: Your Hosting, Our Responsibility!
    cPanel Web Hosting | Reseller Hosting India | VPS | Cloud | Dedicated Server

    Comment


    • #17
      Robot .txt uses for website to tell web robots which page they should crawl on your website, not just only this they also give command to web robot which page they should not crawl and not need to add in their data base.

      Comment


      • #18
        Hello,

        A robots.txt is a text file that are created in the root directory of your website. It gives search engines crawlers instructions as to which pages they can crawl and index, during the crawling and indexing process.
        You can check your robots.txt file: domain name/robots.txt

        Comment


        • #19
          Robots.txt file is used to give permission bots to crawl your website.
          Its a text file in which we can allow Yahoo, Google, MSN to crawl our website. You can allow or disallow bots in robots.txt file.

          Comment


          • #20
            Robots.txt is a text file. In which website owner can declare they want to crawl them website on google or not? Or they can allow and disallow links of them website.

            Comment


            • #21
              robots.txt file to give web robots instructions about their site.
              Ekta Parksville | Rashika Jain | Tarun Trikha

              Comment


              • #22
                A robots. txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

                Comment


                • #23
                  A robots. txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.

                  Comment


                  • #24
                    Robots. txt is a text file webmasters create to instruct web robots how to crawl pages on their website.
                    Viajes a India | Viajes a Rajasthan | Viajes a Agra | Viajes a Mumbai

                    Comment


                    • #25
                      The robots avoidance standard, otherwise called the robots prohibition convention or basically robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard indicates how to advise the web robot about which spaces of the site ought not to be prepared or filtered.

                      Comment


                      • #26
                        Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.

                        Comment


                        • #27
                          RX Marine was established in 1996 in Mumbai, India; as chemical manufacturing company catering exclusively to the needs of the marine industry. In a short span of 12 years the company has established itself as one of the leading wholesale suppliers of a wide range of chemicals for - Marine industry internationally - and other local industires and plants. Our client list bears testimony to this. The RXSOL policy has its foundations on two pillars of strength - a continuous investment in research and development to deliver premium quality products and a commitment to service.
                          Last edited by purnimabiswas; 05-03-2021, 04:18 AM.

                          Comment


                          • #28

                            The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. txt file. The asterisk after “user-agent” means that the robots.

                            Comment


                            • #29
                              Robots.txt is a file that gives instructions to the search engine bot about where it can or can’t go on a website.

                              So every time Google bot crawl a website it looks for the robot.txt file. Before entering into website the bot searches the instruction manual which is in the robot.txt file.
                              MilesWeb: Your Hosting, Our Responsibility!
                              cPanel Web Hosting | Reseller Hosting India | VPS | Cloud | Dedicated Server

                              Comment


                              • #30
                                Gives instructions to search robots which pages of the site should not be crawled.

                                Comment

                                Unconfigured Ad Widget

                                Collapse

                                Announcement

                                Collapse
                                1 of 2 < >

                                FreeHostForum Rules and Guidelines

                                Webmaster forum - Web Hosting Forum,Domain Name Forum, Web Design Forum, Travel Forum,World Forum, VPS Forum, Reseller Hosting Forum, Free Hosting Forum

                                Signature

                                Board-wide Policies:

                                Do not post links (ads) in posts or threads in non advertising forums.

                                Forum Rules
                                Posts are to be made in the relevant forum. Users are asked to read the forum descriptions before posting.

                                Members should post in a way that is respectful of other users. Flaming or abusing users in any way will not be tolerated and will lead to a warning or will be banned.

                                Members are asked to respect the copyright of other users, sites, media, etc.

                                Spam is not tolerated here in most circumstances. Users posting spam will be banned. The words and links will be censored.

                                The moderating, support and other teams reserve the right to edit or remove any post at any time. The determination of what is construed as indecent, vulgar, spam, etc. as noted in these points is up to Team Members and not users.

                                Any text links or images contain popups will be removed or changed.

                                Signatures
                                Signatures may contain up to four lines

                                Text in signatures is subject to the same conditions as posts with respect decency, warez, emoticons, etc.

                                Font sizes above 3 are not allowed

                                Links are permitted in signatures. Such links may be made to non-Freehostforum material, commercial ventures, etc. Links are included within the text and image limits above. Links to offensive sites may be subject to removal.

                                You are allowed ONLY ONE picture(banner) upto 120 pixels in width and 60 pixels in height with a maximum 30kB filesize.

                                In combination with a banner/picture you can have ONLY ONE LINE text link.


                                Advertising
                                Webmaster related advertising is allowed in Webmaster Marketplace section only. Free of charge.

                                Shopping related (tangible goods) advertising is allowed in Buy Sell Trade section only. Free of charge.

                                No advertising allowed except paid stickies in other sections.

                                Please make sure that your post is relevant.


                                More to come soon....
                                2 of 2 < >

                                Advertise at FreeHostForum

                                We offer competitive rates and a many kinds of advertising opportunities for both small and large scale campaigns.More and more webmasters find advertising at FreeHostForum.com is a useful way to promote their sites and services. That is why we now have many long-term advertisers.

                                At here, we also want to thank you all for your support.

                                For more details:
                                http://www.freehostforum.com/threads...eHostForum-com

                                More ad spots:
                                http://www.freehostforum.com/forums/...-FreeHostForum
                                See more
                                See less
                                Working...
                                X