diTii.com Digital News Hub

Sign up with your email address to be the first to know about latest news and more.

I agree to have my personal information transfered to MailChimp (more information)

Oct302009

Google crawls ‘RSS/Atom feeds’ to index URLs

In addition to crawling & indexing web pages via submitted URLs, Google recently launched a feature that uses RSS and Atom feeds to discover new web pages. This helps Google index new webpages faster than traditional methods. As a result, you must make sure that crawling of you feeds isn’t disallowed by your robots.txt. To find out if Googlebot can crawl your feeds, test your feed URLs with robots.txt tester. Going forward, we might also explore mechanisms such as PubSubHubbub to identify updated items, explained Google.

Share This Story, Choose Your Platform!

Do NOT follow this link or you will be banned from the site!