Robotparser doesn't seem to parse correctly

python, python-2.7, robots.txt, web-crawler

Solution

After a few Google searches I didn't find anything about robotparser issue. I ended up with something else, I found a module called reppy which I did a few testing and it seems very powerful. You can install it through pip;

pip install reppy

Here are a few examples (on IPython) using reppy, again, using Google's robots.txt

In [1]: import reppy

In [2]: x = reppy.fetch("http://google.com/robots.txt")

In [3]: x.atts
Out[3]: 
{'agents': {'*': <reppy.agent at 0x1fd9610>},
 'sitemaps': ['http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml',
  'http://www.google.com/hostednews/sitemap_index.xml',
  'http://www.google.com/sitemaps_webmasters.xml',
  'http://www.google.com/ventures/sitemap_ventures.xml',
  'http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml',
  'http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml',
  'http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml',
  'http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml']}

In [4]: x.allowed("/catalogs/about", "My_crawler") # Should return True, since it's allowed.
Out[4]: True

In [5]: x.allowed("/catalogs", "My_crawler") # Should return False, since it's not allowed.
Out[5]: False

In [7]: x.allowed("/catalogs/p?", "My_crawler") # Should return True, since it's allowed.
Out[7]: True

In [8]: x.refresh() # Refresh robots.txt, perhaps a magic change?

In [9]: x.ttl
Out[9]: 3721.3556718826294

In [10]: # It also has a x.disallowed function. The contrary of x.allowed

Problem

I am writing a crawler and for this I am implementing the robots.txt parser, I am using the standard lib robotparser. It seems that robotparser is not parsing correctly, I am debugging my crawler using Google's robots.txt. (Following examples are from IPython) ``` In [1]: import robotparser In [2]: x = robotparser.RobotFileParser() In [3]: x.set_url("http://www.google.com/robots.txt") In [4]: x.read() In [5]: x.can_fetch("My_Crawler", "/catalogs") # This should return False, since it's on Disallow Out[5]: False In [6]: x.can_fetch("My_Crawler", "/catalogs/p?") # This should return True, since it's Allowed Out[6]: False In [7]: x.can_fetch("My_Crawler", "http://www.google.com/catalogs/p?") Out[7]: False ``` It's funny because sometimes it seems to "work" and sometimes it seems to fail, I also tried the same with the robots.txt from Facebook and Stackoverflow. Is this a bug from `robotpaser` module? Or am I doing something wrong here? If so, what? I was wondering if this bug had anything related

Original source