Scrapy - Silently drop an item

Question

I am using Scrapy to crawl several websites, which may share redundant information. For each page I scrape, I store the url of the page, its title and its html code, into mongoDB. I want to avoid duplication in database, thus, I implement a pipeline in order to check if a similar item is already stored. In such a case, I raise a `DropItem` exception. My problem is that whenever I drop an item by raison a `DropItem` exception, Scrapy will display the entire content of the item into the log (stdout or file). As I'm extracting the entire HTML code of each scraped page, in case of a drop, the whole HTML code will be displayed into the log. How could I silently drop an item without its content being shown? Thank you for your time! ``` class DatabaseStorage(object): """ Pipeline in charge of database storage. The 'whole' item (with HTML and text) will be stored in mongoDB. """ def __init__(self): self.mongo = MongoConnector().collection def process_item(self, item, spider): """ Method in ch…

Accepted Answer

The proper way to do this looks to be to implement a custom LogFormatter for your project, and change the logging level of dropped items.

Example:

```
from scrapy import log
from scrapy import logformatter

class PoliteLogFormatter(logformatter.LogFormatter):
    def dropped(self, item, exception, response, spider):
        return {
            'level': log.DEBUG,
            'format': logformatter.DROPPEDFMT,
            'exception': exception,
            'item': item,
        }
```

Then in your settings file, something like:

```
LOG_FORMATTER = 'apps.crawler.spiders.PoliteLogFormatter'
```

I had bad luck just returning "None," which caused exceptions in future pipelines.

Scrapy - Silently drop an item

Solution

Problem