Google have released a new user agent for robots.txt called “Googlebot-News,” that gives publishers even more control over their content. In addition to contacting via form; Now, publishers can manage their content in Google News with just adding Googlebot-News specific directives to their robots.txt file. Similar to Googlebot and Googlebot-Image user agents, Googlebot-News user agent can be used to specify which pages of a website should be crawled and ultimately appear in Google News. Here’re few examples:
Include pages in both Google web search and News:
User-agent: Googlebot
Disallow:Include pages in Google web search, but not in News:
User-agent: Googlebot
Disallow:User-agent: Googlebot-News
Disallow: /Include pages in Google News, but not Google web search:
User-agent: Googlebot
Disallow: /User-agent: Googlebot-News
Disallow:Block different sets of pages from Google web search and Google News:
User-agent: Googlebot
Disallow: /latest_newsUser-agent: Googlebot-News
Disallow: /archivesStop Google web search and Google News from crawling pages:
User-agent: Googlebot
Disallow: /
Recommend this story
Email Newsletter
Missing out on the latest diTii.com news? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before your subscription is activated - please check your spam folder if you don't receive this.
About the AuthorDG