June 4, 2008
2:31 am | Last updated: June 4, 2008 at: 2:56 am
Robots Exclusion Protocol (REP) improvements
Learn more: You can find out more about robots.txt at http://www.robotstxt.org and at Google's Webmaster help center, which contains lots of helpful information, including:
- How to create a robots.txt file
- Descriptions of each user-agent that Google uses
- How to use pattern matching
- How often we recrawl your robots.txt file
We've also done several posts in our webmaster blog about robots.txt that you may find useful, such as:
There is also a useful list of the bots used by the major search engines. Also see blog posts published by Yahoo! and Microsoft.
Loading
No followup yet
Leave a Response
Leave a comment »