A web crawler is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.