The Internet is so huge that search engines bots can crawl forever discovering new links. Webmasters can help the search engines and themselves by removing useless and duplicate content from the crawler’s path – make sure “less is more.”
In a blog post, on Bing Webmaster Center, Duane Forrester, says that to fix duplicate content problems on a site, relying on canonical tag is not necessarily the perfect solution — and suggests a better solution to fix the duplicate problems than simply relying only on the canonical tag.
Forrester explains that “search engines will follow links to canonical destination URLs discovered in the canonical source URLs, but they’ll continue to visit canonical source URLs as well. By visiting canonical source pages, search engines waste the limited crawl bandwidth they generally have assigned to a website, and worse, may continue discovering inside the canonical source page plenty of URLs with extra useless URLs parameters.” “This extra-crawling and processing impact the overall quality of the site in our index and can create unneeded crawl loads on websites,” Forrester adds.
So what is the solution? Forrester suggests, if you’ve duplicate problems due to extra URLs parameters, using the URL Normalization feature in the Bing Webmaster Tools is the preferred method, that helps you direct bingbot which parameters can go away.
“By normalizing, bing crawler willn’t visit the URLs with extra parameters except for an occasional test of the quality of the normalization rules,” he said. “This will benefit webmasters through less crawling which reduces server loads, fresher copy in bing index of the canonical destination URLs and less out-links being discovered with extra-parameters,” said Forrester.
As an advantage, you can continue using the canonical tag on these pages as complimentary information to the URL Normalization rules provided. Also, URL Normalization can be done without any code change and can also help in http 301 redirection instead of using the canonical tag when possible.
To help, users, Bing provide in the tool the most common duplicate URL parameters detected on a site. And, also let you review what Bing index via the Index Explorer feature to see any other parameters you’d like to skip when crawling.