Enabling deltafetch in scrapy

Question

I have worked on `scrapy` a bit and now I have my spider ready. But now I want my spider to scrape only those items which is not been scraped in its previous run, and scrape only the new contents. By achieving this I can reduce the runtime of my spider. While studying about this I came across deltafetch, Which I think will serve my requirement. But I am not being able to import that feature. I would be glad if any body could guide me about using it in a well defined way. And also if there is any other middleware which serve the similar purpose I would be interested to know.

Accepted Answer

deltafetch is part of the scrapylib library rather than the default scrapy package so I think that's why you can't import it. Here's how I got it to work:

First create a new directory in your main project module (where your spiders directory, items.py ect. are) called middlewares.

Then inside it put an empty `__init.py__` and the deltafetch.py file from github.

In your settings.py put :

```
SPIDER_MIDDLEWARES = {
    'yourprojectname.middlewares.deltafetch.DeltaFetch': 100,
}

DELTAFETCH_ENABLED = True
```

There are more available settings but those are documented in the doc string of the `DeltaFetch` class. Hope that helps!

Enabling deltafetch in scrapy

Solution

Problem