Google now offers an option to web publishers called “First Click Free” with two main goals: “To include highly relevant content in Google's search index. This provides a better experience for Google users who may not have known that content existed. Secondly, to provide a promotion and discovery opportunity for publishers with restricted content.”
First Click Free is designed to protect your content while allowing you to include it Google's search index. To implement you must:
- you must allow all users who find your page through Google search to see the full text of the document
- site content must be identical to the content that is shown to Googlebot
- users must be able to view multi-page article (if any)
- user's first click to your content is free and does not require logging in
- you may, however, block the user with a login or payment or registration request when he tries to click away from that page to another section of your content site.
- to include restricted content in Google's search index, Google crawler must be allowed access behind registration or login forms. You need to configure your website to serve the full text of each document when the request is identified as coming from Googlebot via the user-agent and IP-address. It's equally important that your robots.txt file allows access of these URLs by Googlebot.
- If the “Referer” HTTP request-header field has Google domain, like www.google.com or www.google.de, your site must display full content instead of protected version, when users click a Google search result to access your content.