The search system robots (bots) are programs belonging to search engine software.
The search system robots are occupied with indexing of pages – they download pages and put them in the index. The smart search system robot can determine whether the page has been changed since the last visit and is it worth spending time and computer resources for repeat indexing of a page.
