Web Crawler

The function of a web crawler is to automatically browse the web in search of web pages to analyze. Using well-defined criteria, the web crawler acts autonomously and automatically adjusts its task.

Google in particular uses them to update its indexes.