Robots Exclusion Protocol (REP) improvements

Robots Exclusion Protocol (REP), lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site via robots.txt or down to a much more granular level for individual pages (via META tags). Google Webmaster Centeral Blog released detailed documentation about implementing REP to provide a common implementation for webmasters and make it easier […]

Robots Exclusion Protocol (REP), lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site via robots.txt or down to a much more granular level for individual pages (via META tags). Google Webmaster Centeral Blog released detailed documentation about implementing REP to provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.

Common REP Directives: The following list are all the major REP features currently implemented by Google, Microsoft, and Yahoo!. With each feature, you'll see what it does and how you should communicate it.