Avoid Duplicate URL Crawling

scrapy

Solution

I believe what you are looking for is "persistence support", to pause and resume crawls.

To enable it you can do:

scrapy crawl somespider -s JOBDIR=crawls/somespider-1

You can read more about it here.

Problem

I coded a simple crawler. In the settings.py file, by referring to scrapy documentation, I used ``` DUPEFILTER_CLASS = 'scrapy.dupefilter.RFPDupeFilter' ``` If I stop the crawler and restart the crawler again, it is scraping the duplicate urls again. Am I doing something wrong?

Original source