Is Scrapy single-threaded or multi-threaded?
multithreading, python, scrapy, web-crawler
Solution
Scrapy is single-threaded, except the interactive shell and some tests, see source.
It's built on top of Twisted, which is single-threaded too, and makes use of it's own asynchronous concurrency capabilities, such as `twisted.internet.interfaces.IReactorThreads.callFromThread`, see source.
Problem
There are few concurrency settings in Scrapy, like CONCURRENT_REQUESTS. Does it mean, that Scrapy crawler is multi-threaded? So if I run `scrapy crawl my_crawler` it will literally fire multiple simultaneous requests in parallel? Im asking because, I've read that Scrapy is single-threaded.