Google News new user agent Googlebot-News' for robots.txt

Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent […]

Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News. Here’re few examples:

Include pages in both Google web search and News:
User-agent: Googlebot
Disallow:

Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-News
Disallow: /

Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-News
Disallow:

Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_news

User-agent: Googlebot-News
Disallow: /archives

Stop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /