Google News new user agent Googlebot-News' for robots.txt

Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent […]

Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News. Here’re few examples:

Include pages in both Google web search and News:
User-agent: Googlebot
Disallow:

Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-News
Disallow: /

Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-News
Disallow:

Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_news

User-agent: Googlebot-News
Disallow: /archives

Stop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /

About The Author

Deepak Gupta is a IT & Web Consultant. He is the founder and CEO of diTii.com & DIT Technologies, where he's engaged in providing Technology Consultancy, Design and Development of Desktop, Web and Mobile applications using various tools and softwares. Sign-up for the Email for daily updates. Google+ Profile.