Google Webmaster Tools’ Crawl errors feature today received some significant enhancements, that now lets you focus on fixing what’s most important first. “We now detect and report many new types of errors. To help make sense of the new data, we’ve split the errors into two parts: site errors and URL errors,” informs Google.
“We’ve ranked the errors so that those at the top of the priority list will be ones where there’s something you can do, whether that’s fixing broken links on your own site, fixing bugs in your server software, updating your Sitemaps to prune dead URLs, or adding a 301 redirect to get users to the “real” page,” Google said.
Once the issues are fixed, you can mark the error as “fixed” to help Google remove the error from the list. “The errors once marked as fixed won’t be included in the top errors list, unless we’ve encountered the same error when trying to re-crawl a URL,” said Google.
Here is what is new?
Site Errors affect entire site–includes DNS resolution failures, connectivity issues with web server, and problems fetching robots.txt file. Until now, Google was reporting these errors by URL, but now Google say they keep track of the failure rates for each type of site-wide error. And, will also send alerts when these errors become frequent enough that they warrant attention.
URL errors are specific to a particular page, Google now break the URL errors down into various categories based on what caused the error. “If your site serves up Google News or mobile (CHTML/XHTML) data, we’ll show separate categories for those errors,” Google infomred.
Less is more
Until now, GWT was showing at most 100,000 errors of each type. In this version, Google will only give the most important errors up front. “For each category, we’ll give you what we think are the 1000 most important and actionable errors. You can sort and filter these top 1000 errors, let us know when you think you’ve fixed them, and view details about them,” informed Google.
For sites with more than 1000 errors of a given type, will still be able to see the total number of errors of each type, as well as a graph showing historical data going back 90 days.
Those who worry that 1000 error details plus a total aggregate count will not be enough, Google says they would be adding programmatic access (an API) to allow you to download every last error you have.
Google also removed the list of pages blocked by robots. Instead, Google will now provide information about roboted URLs in the “Crawler access” feature under “Site configuration”.
Dive into the details
Now, clicking an individual error URL from the main list brings up a detail pane with additional information, including when we last tried to crawl the URL, when we first noticed a problem, and a brief explanation of the error.
“From the details pane you can click on the link for the URL that caused the error to see for yourself what happens when you try to visit it. You can also mark the error as “fixed” (more on that later!), view help content for the error type, list Sitemaps that contain the URL, see other pages that link to this URL, and even have Googlebot fetch the URL right now, either for more information or to double-check that your fix worked,” Google explained.