Robots.txt is a file or files, within a site, that suggests which parts of your website search engines such as Google should and shouldn’t crawl.