robots.txt - exclude any URL that contains "/node/"
robots.txt, seo
Solution
`Disallow: /node/` will disallow any url that starts with `/node/` (after the host). The asterisk is not required.
So it will block www.mysite.com/node/bar.html, but will not block www.mysite.com/foo/node/bar.html.
If you want to block anything that contains `/node/`, you have to write `Disallow: */node/`
Note also that Googlebot can cache robots.txt for up to 7 days. So if you make a change to your robots.txt today, it might be a week before Googlebot updates its copy of your robots.txt. During that time, it will be using its cached copy.
Problem
How do I tell crawlers / bots not to index any URL that has /node/ pattern? Following is since day one but I noticed that Google has still indexed a lot of URLs that has /node/ in it, e.g. www.mywebsite.com/node/123/32 Disallow: /node/ Is there anything that states that do not index any URL that has /node/ Should I write something like following: Disallow: /node/* Update: The real problem is despite: Disallow: /node/ in robots.txt, Google has indexed pages under this URL e.g. www.mywebsite.com/node/123/32 /node/ is not a physical directory, this is how drupal 6 shows it's content, I guess this is my problem that node is not a directory, merely part of URLs being generated by drupal for the content, how do I handle this? will this work? Disallow: /*node Thanks