Google News Now Using Googlebot For Crawling

Google News now crawl with Google's primary user-agent, Googlebot. With that, "any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot," informs David Smydra, Google […]

Google News now crawl with Google's primary user-agent, Googlebot. With that, "any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot," informs David Smydra, Google News Product Specialist.

Although Google's Help Center provides detailed guidance on using the robots exclusion protocol for Google News, and publishers can contact the Google News Support Team if they have any questions, the team explains:

  • "Although you'll now only see the Googlebot user-agent in your site's logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies. (You can always check whether your site is included in Google News by searching with the "site:" operator.
  • Your analytics tool will still be able to differentiate user traffic coming to your website from Google Search and traffic coming from Google News, so you should see no changes there. The main difference is that you will no longer see occasional automated visits to your site from the Googlebot-news crawler.
  • If you're currently respecting webmaster guidelines for Googlebot, you'll not need to make any code changes to your site. Sites that have implemented subscriptions using a metered model or who've implemented First Click Free willn't experience any changes. For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page. Our Webmaster Guidelines provide additional information about "cloaking" (i.e., showing a bot a different version than what users experience). Learn more about Google News and subscription publishers in this Help Center article.
  • Rest assured, your Sitemap will still be crawled.
  • Sitemap and are interested in getting started, please follow this link.
  • For any publishers that wish to opt out of Google News and stay in Google Search, you can simply disallow Googlebot-news and allow Googlebot. For more information on how to do this, consult our Help Center," the Google News team explained.