What is a web crawler?

[markusgomes60]

Many of the website owners, SEO experts and forum members have been discussing about a web crawler. They consider it to be the most important element for the website owner. Is it a computer program? How does it help the sites? Does it help in browsing the World Wide Web in a methodical, automated manner? I heard that the web crawlers are also termed as ants, automatic indexers, bots, and worms. Is that true? Will this assist the website to be visible on the search engine? Can I expect the search engine spider to crawl the web pages of my site?

[ruby.mein]

Web crawler is a computer program that browses the web in a methodical, automated manner or in an orderly fashion. Sometimes called Spiders. Spiders also it identifies all the hyperlinks in the page and adds them to the list of URLs to visit.

[rockypontingg]

Website owners, SEO experts and many of the members of the forum discussion about crawler.Web a web crawler is a computer program that a systematic, automated manner or in an orderly fashion web browses.

[kinah1]

It is a machine that visit your site everyday.

[billybrwon]

it can track your record and direct you when you require them another time. it can load the web pages as very quickly.

[sourecdamine]

It is a computer program that surfs the World Wide Web in a methodical way, automatically. Other terms for web crawlers are ants, automatic indexers, bots and worms or spider, web robot, or specifically in the FOAF community SCUTT web.

[lapssone]

I just by a spider crawl sitemap to provide a platform for

[sakthi]

A web crawler is a relatively simple automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of the data it's looking for. Alternative names for a web crawler include web spider, web robot, bot, crawler, and automatic indexer.

............................

[TinaTinka]

The web crawler is really a program which visits Internet sites and scans their pages along with other information to be able to create records for the search engines index.

[jainsumit]

web crawler are search engine bots which collects the data from web and use that data to display results in Searching. The Google bot is the name of google web crawler.

[hgtech04]

It's Google Spider

[jiyaalbert]

web crawler is a computer program which crawl your site every day to create record for search engine.

[ankus]

Web crawler is an internet bot which systematically browses the world wide web, typically for the purpose of web indexing.

[Sallynguyen]

Many of the website owners, SEO experts and forum members have been discussing about a web crawler. They consider it to be the most important element for the website owner. Is it a computer program? How does it help the sites? Does it help in browsing the World Wide Web in a methodical, automated manner? I heard that the web crawlers are also termed as ants, automatic indexers, bots, and worms. Is that true? Will this assist the website to be visible on the search engine? Can I expect the search engine spider to crawl the web pages of my site?

Search engine for locating information on www, it indexes all the words in a document, adds them to a database, then follow all hyperlinks and indexes and adds that information also to the database. Also called web spider or spider.

[sonipat]

Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code.