Robots.txt for multiple domains
seo
Solution
The robots.txt can only inform the search engines of sitemaps for its own domain. So that one will be the only one it honors when it crawls that domain's robots.txt. If all three domains map to the same website and share a robots.txt then the search engines will effectively find each sitemap.
Problem
We have different domains for each language - www.abc.com - www.abc.se - www.abc.de And then we have different sitemap.xml for each site. In robots.txt, I want to add sitemap reference for each domain. - Is it possible to have multiple sitemap references for each domain in single robots.txt? - If there are multiple, which one does it pick?