Crawl-delay intro to reduce server load for MSNBot

The Redmond company began introducing the new release of Live Search family of services and solutions  last week. According to Nathan Buggia, Live Search Webmaster Center, the enhancements spanning across various items in the Windows Live catalog also affect the crawler associated with Live Search. “A few months ago we announced two new features to MSNBot […]

The Redmond company began introducing the new release of Live Search family of services and solutions  last week. According to Nathan Buggia, Live Search Webmaster Center, the enhancements spanning across various items in the Windows Live catalog also affect the crawler associated with Live Search.

“A few months ago we announced two new features to MSNBot to reduce the burden of crawling on your website. These were part of a series of improvements we’re making to our crawler during the Spring to increase the freshness and breadth of content in our index. As part of these latest improvements, you may notice an increase in the amount of traffic from MSNBot starting over the next couple weeks,” Buggia said.

In terms of search, Microsoft continues to be the bronze winner in the three horse race against Yahoo and Google. However, the Redmond company is not yet ready to throw down the towel. And while it is waiting for Yahoo to accept its $44.6 billion dollar unsolicited acquisition proposal, Microsoft continues to hammer away at its own product. The introduction of features such as HTTP Conditional GET, HTTP Compression and now Crawl-delay are illustrative examples of the efforts poured by Microsoft into Live Search and the crawler associated with the search engine.

“Setting the crawl delay reduces the load on your servers, but it also increases the amount of time it will take MSNBot to index your website (proportional to the length of the delay), and possibly make it more difficult for your customers to find your site on Live Search,” Buggia added.

Also, if you feel that MSNBot is crawling your site too frequently, you can use the crawl delay directive in robots.txt. Please refer to the MSNBot support page for more information. Here are a few recommended settings:

Slow (wait 5 seconds between each request)

Crawl-delay: 5

Really Slow (wait 10 seconds between each request)

Crawl-delay:  10

Note that setting the crawl delay reduces the load on your servers, but it also increases the amount of time it will take MSNBot to index your website (proportional to the length of the delay), and possibly make it more difficult for your customers to find your site on Live Search.

Another great way to reduce the impact of MSNBot on your website is to enable HTTP Conditional GET and HTTP Compression as outlined in our prior blog post.

Microsoft, Windows Live, Windows Live Search, MSNBot, Search Engine, Crawling, Indexing, Crawl-delay