Python Scrapy - Direct spider to specific Pipeline
python-2.7, scrapy
Solution
`ITEM_PIPELINES` setting is defined globally for all spiders in the project during the engine start. It cannot be changed per spider on the fly.
Here's what you can do. Define what spiders should be processed via the pipeline in the pipeline itself. Skip/continue processing items returned by spiders in the process_item method of your pipeline, e.g.:
def process_item(self, item, spider):
if spider.name not in ['spider1', 'spider2']:
return item
# process item
Also see:
- Is there any method to using seperate scrapy pipeline for each spider?
Hope that helps.
Problem
I have a Scrapy project with multiple spiders along with multiple pipelines. Is there a way I can tell Spider A to use pipeline A, etc??? My pipelines.py has multiple pipeline classes each doing something different and I want to be able to tell a spider to use a specific pipeline. I do not see any obvious ways looking at the available scrapy commands to do this...