How Scrapy filters the crawled urls?
scrapy
Solution
By default scrapy keep a fingerprint of seen requests. This list is kept in memory in a python set and appended a file call requests.seen in the directory defined by the JOBDIR variable. If you restart scrapy the file is reloaded into the python set. The class that control this is in scrapy.dupefilter You can overload this class if you need a different behaviour.
Problem
I want to know how Scrapy filters those crawled urls? Does it store all urls which are crawled in something like `crawled_urls_list`, and when it get a new url it looks up the list to check if the url exists ? Where are the codes of this filtering part of CrawlSpider(/path/to/scrapy/contrib/spiders/crawl.py) ? Thanks a lot!