Separate output file for every url given in start_urls list of spider in scrapy

Question

I want to create separate output file for every url I have set in start_urls of spider or somehow want to split ouput files start url wise. Following is the start_urls of my spider ``` start_urls = ['http://www.dmoz.org/Arts/', 'http://www.dmoz.org/Business/', 'http://www.dmoz.org/Computers/'] ``` I want to create separate output file like Arts.xml Business.xml Computers.xml I don't know exactly how to do this. I am thinking to achieve this by implementing some thing like following in spider_opened method of item pipeline class, ``` import re from scrapy import signals from scrapy.contrib.exporter import XmlItemExporter class CleanDataPipeline(object): def __init__(self): self.cnt = 0 self.filename = '' @classmethod def from_crawler(cls, crawler): pipeline = cls() crawler.signals.connect(pipeline.spider_opened, signals.spider_opened) crawler.signals.connect(pipeline.spider_closed, signals.spider_closed) return pipeline def spider_opened(self, spider): referer_url = response.request.he…

Accepted Answer

I'd implement a more explicit approach (not tested):

configure list of possible categories in `settings.py`:

```
CATEGORIES = ['Arts', 'Business', 'Computers']
```

define your `start_urls` based on the setting

```
start_urls = ['http://www.dmoz.org/%s' % category for category in settings.CATEGORIES]
```

add `category` `Field` to the `Item` class

in the spider's parse method set the `category` field according to the current `response.url`, e.g.:

```
def parse(self, response):
     ...
     item['category'] = next(category for category in settings.CATEGORIES if category in response.url)
     ...
```

in the pipeline open up exporters for all categories and choose which exporter to use based on the `item['category']`:

```
def spider_opened(self, spider):
    ...
    self.exporters = {}
    for category in settings.CATEGORIES:
        file = open('output/%s.xml' % category, 'w+b')
        exporter = XmlItemExporter(file)
        exporter.start_exporting()
        self.exporters[category] = exporter

def spider_closed(self, spider):
    for exporter in self.exporters.itervalues(): 
        exporter.finish_exporting()

def process_item(self, item, spider):
    self.exporters[item['category']].export_item(item)
    return item
```

You would probably need to tweak it a bit to make it work but I hope you got the idea - store the category inside the `item` being processed. Choose a file to export to based on the item category value.

Hope that helps.

Separate output file for every url given in start_urls list of spider in scrapy

Solution

Problem