The robots.txt file instructs search engine crawlers about which pages or files they should or shouldn’t request from a site