Googlebot and infinite space

Infinite space are very large numbers of links that usually provide little or no new content for indexing. When Googlebot crawls the web, and if this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site. The classic example of an […]

Infinite space are very large numbers of links that usually provide little or no new content for indexing. When Googlebot crawls the web, and if this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site.

The classic example of an “infinite space” is a calendar with a “Next Month” link. It may be possible to keep following those “Next Month” links forever! Of course, that's not what you want Googlebot to do.

Google started notifying site owners when discoverd this problem in the Message Center of Webmaster Tools. Google Webmaster Tools Help article describes more ways infinite spaces can arise, and provides recommendations on how to avoid the problem. One fix is to eliminate whole categories of dynamically generated links using your robots.txt file.