At SMX West the search engines made an announcement on the Search Engineers Q&A panel about an update to the sitemaps.org protocol. The new update enables site owners to store their XML Sitemap files in any location, even on a different domain than the one referenced in the Sitemap. A welcome change for those who manage multiple domains and would like to keep all Sitemap files in one place, as well as for those who would like to store their Sitemap in a location other than the root. The only caveat is, you have to edit the robots.txt file of the domain the Sitemap file references.
How this works and how to implement it on your site? Earlier, your Sitemap file had to be stored in the same location as the URLs listed in Sitemap. For instance:
http://www.example.com/sitemap.xml could include:
But it could not include:
Similarly, http://www.example.com/folder1/sitemap.xml could include:
And could not include:
The protocol required this to ensure that whoever could modify the Sitemap file also had ownership rights of the URLs listed in the Sitemap. However, this setup was problematic for some sites, including those with multiple domains and subdomains who would found it difficult or technically infeasible to write Sitemap files to multiple locations. Some sites also found it technically different to write Sitemap files to the root of the domain.
With Google, site owners can overcome this issue by verifying ownership of all domains in Webmaster Centralverifying ownership of all domains in Webmaster Central, then submitting Sitemaps for any of those domains from any location. However, as this solution only works for Google, it's not useful for those who want one process for generating Sitemaps for submission to all engines. (Which, is, after all, the spirit of sitemaps.org.)
With the latest sitemaps.org update:
- Each Sitemap file can reference only one domain.
- The Sitemap can be placed anywhere (on a different domain or in a subfolder) as long as its location is referenced in the domain's robots.txt file.
Referencing The Sitemap File In robots.txt: Simply point at the Sitemap location in your robots.txt file. For instance, if you have a Sitemap file for www.example and you want to store it at www.example1.com/sitemap.xml, add to the following line to www.example.com/robots.txt:
This line lets the search engines know that you have ownership of www.example.com (since you can modify the robots.txt file for it) and that the Sitemap for that domain is located at http://www.example1.com/sitemap.xml.
If you want to store Sitemaps for multiple domains in one location, simply create a Sitemap for each domain, then modify the robots.txt for each of them to point at the Sitemap file location as shown in right pic:
The same technique works if you want to store your Sitemap in a location other than the root. If you want to store the Sitemap at http://www.example.com/sitemap/sitemap.xml, simply modify the robots.txt file for www.example.com with the following:
For more information, see the sitemaps.org documentation or each search engine's blog post:
SMX, Sitemap, Robots.txt, Search Engine, SEO, XML, Update