Live Search (MSNbot) crawler issues: Update

A follow up is posted to the last month complaints about the MSNbot crawling issues: “Last week, a patch to the crawler was deployed, which should significantly reduce the problems.  Interestingly, while part of the problem involved “cloaking detection”, where websites try to trick search engines into better positioning, some of the problems were also caused by attempts to […]
A follow up is posted to the last month complaints about the MSNbot crawling issues: “Last week, a patch to the crawler was deployed, which should significantly reduce the problems.  Interestingly, while part of the problem involved “cloaking detection”, where websites try to trick search engines into better positioning, some of the problems were also caused by attempts to gain much quicker RSS feed information.[…] A crawl-delay: 5 means we wait 5 seconds per page request. That means if you have a site with 100 pages, it will take us 8.3 minutes to crawl your entire site. A site with 1,000 pages will take 83 minutes.[…]”

About The Author

Deepak Gupta is a IT & Web Consultant. He is the founder and CEO of diTii.com & DIT Technologies, where he's engaged in providing Technology Consultancy, Design and Development of Desktop, Web and Mobile applications using various tools and softwares. Sign-up for the Email for daily updates. Google+ Profile.