The search engine will put this processed information into the search engine’s index library. The following focuses on the Google search engine.
The Google search engine uses two crawlers to crawl web page content, namely: Freshbot and Deepbot. The deep crawler (Deepbot) is executed once a month, and the content it visits is in Google’s main index, while the refresh crawler (Freshbot) discovers new information and resources on the Internet day and night, and then frequently visits and updates. Because, generally, the first or relatively new websites discovered by Google are visited in the Freshbot list.
The results of Freshbot are stored in another separate database. Since Freshbot works non-stop and refreshes the access content, the web pages it discovers or updates will be rewritten when it is executed. And these contents provide search results together with Google’s main indexer. Previously, some websites were included by Google at the beginning, but after a few days, this information disappeared from Google’s search results. Until a month or two later, the results reappeared in Google’s main index. This is because Freshbot is constantly updating and refreshing content, while Deepbot only attacks once a month, so the results in Freshbot have not yet been updated to the main index and are replaced by new content. It is not until Deepbot revisits this page and collects it that it actually enters Google’s main index database.