Last week, a group of newspaper and magazine publishers signed a declaration stating that "Universal access to websites does not necessarily mean access at no cost," and that they "no longer wish to be forced to give away property without having granted permission." "More than 25,000 news organizations across the globe choose to make their content available in Google News and other web search engines. If at any point a web publisher wants to stop indexing their content, they're able to do so quickly and effectively," News Blog. To stop google bot from crawling a specific page, just add '<meta name="googlebot" content="noindex">' to the page. And, to stop indexing entire site, add following two lines in Robots Exclusion Protocol (REP) to deny permission:

User-agent: *
Disallow: /

More info: European Public Policy Blog