Bing crawler (MSNBot) Crawl delay

Search engines, such as Bing, need to regularly crawl websites not only to index new content, but also to check for content changes and removed content. Larger sites that’ve a great many pages of content may need to be crawled more deeply and/or more often so that their latest content may be added into the […]

Search engines, such as Bing, need to regularly crawl websites not only to index new content, but also to check for content changes and removed content. Larger sites that’ve a great many pages of content may need to be crawled more deeply and/or more often so that their latest content may be added into the index. Many factors affect the crawling of a site, including (but not limited to): total number of pages on a site, size of the content (PDFs and Microsoft Office files’re typically much larger than regular HTML files), freshness of the content, number of allowed concurrent connections, bandwidth of the site, and how highly does the site rank. Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues via directives of the Robots Exclusion Protocol (REP) as listed in a site’s robots.txt file, which is stored at the root folder of a website. The robots.txt file is the only valid place to set a crawl-delay directive for MSNBot:

User-agent: *
Crawl-delay: 1