Automated website crawlers are powerful tools to help crawl and index content on the web. As a webmaster, you may wish to guide them towards your useful content and away from irrelevant content.
Google has published a online document entitled “Controlling crawling and indexing”, that provides lots of other information about controlling the crawling and indexing of your site. They consist of the robots.txt file to control crawling, as well as the robots meta tag and X-Robots-Tag HTTP header element to control indexing. The robots.txt standard predates Google and is the accepted method of controlling crawling of a website.