diTii.com Digital News Hub

Sign up with your email address to be the first to know about latest news and more.

I agree to have my personal information transfered to MailChimp (more information)

Dec022009

Google News new user agent Googlebot-News’ for robots.txt

Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News. Here’re few examples:

Include pages in both Google web search and News:
User-agent: Googlebot
Disallow:

Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-News
Disallow: /

Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-News
Disallow:

Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_news

User-agent: Googlebot-News
Disallow: /archives

Stop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /

Share This Story, Choose Your Platform!

Get Latest News

Subscribe to Digital News Hub

Get our daily newsletter about the latest news in the industry.
First Name
Last Name
Email address
Secure and Spam free...