A follow up is posted to the last month complaints about the MSNbot crawling issues: “Last week, a patch to the crawler was deployed, which should significantly reduce the problems.  Interestingly, while part of the problem involved “cloaking detection”, where websites try to trick search engines into better positioning, some of the problems were also caused by attempts to gain much quicker RSS feed information.[…] A crawl-delay: 5 means we wait 5 seconds per page request. That means if you have a site with 100 pages, it will take us 8.3 minutes to crawl your entire site. A site with 1,000 pages will take 83 minutes.[…]”