Following a recent complaint of a Google AdWords advertiser that Bing had discovered, crawled and indexed their AdWords tracking URLs --- leading to inflated AdWords clicks and conversions.
"I noticed something shocking... The URLs listed on Bing for 5 of the 10 magazine pages were from AdWords. That means that Microsoft is using AdWords for page discovery!," said Allmagazineprices.
Allmagazineprices notes, that despite setting up and running the same ad campaign on Microsoft's adCenter, and Google's AdWords -- Bingbot had crawled their AdWords listing. Had, Bing crawled its own paid search listings, the website could have saved inflated clicks.
Google notes, "Invalid clicks can be generated either by humans or technological means, including various types of deceptive software programs, such as scripts or bots."
Today, Microsoft has confirmed that they do discover and index URLs that they find through users surfing the Internet with the Bing Toolbar installed. Duane Forrester, a Senior Product Manager at Microsoft said:
"Yes, as with some other toolbars, the Bing toolbar (when permitted by the user) may record the open (not https) web sites that a user visited to add to our knowledge of the internet in order to improve the search results we provide to users. This in turn (if the site permits) may indeed lead to our crawling links we have discovered. As noted, any site can control the indexing of pages via the NOINDEX meta tag."
This can be a concern for those who know certain URLs are only accessible if the individual knows about it and navigates to the URL with the Bing Toolbar installed. "For example, YouTube has a setting where you can share a video privately with someone else by giving them a special URL. If the user they are sharing that URL with has the Bing Toolbar installed, Bing can technically discover the URL, assuming the page is not blocked via the NOINDEX meta tag or other means," explains SEL.
Duane said, "for the case of Advertisers, they can (and often do) notify crawlers not to crawl their click tracking URLs by using robots.txt disallow rules, Bingbot honors these rules and will not index the URL nor follow the redirection."