Bots, also sometimes referred to as ‘spiders’ or crawlers’, is what search engines use to read content across the web and index them on their search engine results pages.