Robots Exclusion Protocol (REP) improvements

Robots Exclusion Protocol (REP), lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site via robots.txt or down to a much more granular level for individual pages (via META tags). Google Webmaster Centeral Blog released detailed documentation about implementing REP to provide a common implementation for webmasters and make it easier […]

Robots Exclusion Protocol (REP), lets publishers control how search engines access their site: whether it's controlling the visibility of their content across their site via robots.txt or down to a much more granular level for individual pages (via META tags). Google Webmaster Centeral Blog released detailed documentation about implementing REP to provide a common implementation for webmasters and make it easier for any publisher to know how their REP directives will be handled by three major search providers -- making REP more intuitive and friendly to even more publishers on the web.

Common REP Directives: The following list are all the major REP features currently implemented by Google, Microsoft, and Yahoo!. With each feature, you'll see what it does and how you should communicate it.

About The Author

Deepak Gupta is a IT & Web Consultant. He is the founder and CEO of diTii.com & DIT Technologies, where he's engaged in providing Technology Consultancy, Design and Development of Desktop, Web and Mobile applications using various tools and softwares. Sign-up for the Email for daily updates. Google+ Profile.