Running scrapy from script not including pipeline

Question

I'm running scrapy from a script but all it does is activate the spider. It doesn't go through my item pipeline. I've read http://scrapy.readthedocs.org/en/latest/topics/practices.html but it doesn't say anything about including pipelines. My setup: ``` Scraper/ scrapy.cfg ScrapyScript.py Scraper/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py my_spider.py ``` My script: ``` from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy.settings import Settings from scrapy import log, signals from Scraper.spiders.my_spider import MySpiderSpider spider = MySpiderSpider(domain='myDomain.com') settings = get_project_settings crawler = Crawler(Settings()) crawler.signals.connect(reactor.stop, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) crawler.start() log.start() log.msg('Reactor activated...') reactor.run() log.msg('Reactor stopped.') ``` My pipeline: ``` from scrapy.exceptions import DropItem from scrapy import log imp…

Accepted Answer

You need to actually call get_project_settings, Settings object that you are passing to your crawler in your posted code will give you defaults, not your specific project settings. You need to write something like this:

```
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
crawler = Crawler(settings)
```

Running scrapy from script not including pipeline

Solution

Problem