Should I use different case-spellings for case-insensitive directories in robots.txt?
case-sensitive, robots.txt, web-crawler
Solution
The original robots.txt specification doesn't say anything about typecase in file paths, but according to Google's robots.txt specification, file paths are definitely case-sensitive. Google clearly states that "Disallow: /img/" only blocks "/img/", not "/Img/" or "/IMG/". Your solution is definitely valid, and will solve the problem.
That being said, I would only resort to this solution if I had reason to believe the alternate-case URLs were actually being crawled, and they were causing a problem. You can easily turn your robots.txt file into an unmaintainable mess otherwise.
Problem
Unfortunately, I’ve got case-insensitive servers that cannot be replaced in the short term. Some directories need to be excluded from crawling, so I have to `Disallow` them in my `robots.txt`. Let’s take `/Img/` as example. If I keep it all lower case… ``` User-agent: * Disallow: /img/ ``` … it does not map to the actual physical path, and addresses with `/Img/` or `/IMG/` are not applied the `Disallow` directive. Crawlers will treat these variations as distinct paths. It’s fun to look at Microsoft’s robots.txt in this matter. They probably use IIS servers, and SERPs are just full of disallowed addresses–only with other cases. What can I do? Is it valid (and effectual) to state the following? ``` User-agent: * Disallow: /Img/ Disallow: /img/ Disallow: /IMG/ ```