How to make Scrapy show user agent per download request in log?
python, scrapy, user-agent, web-crawler, web-scraping
Solution
Just FYI.
I've implemented a simple `RandomUserAgentMiddleware` middleware based on `fake-useragent`.
Thanks to `fake-useragent`, you don't need to configure the list of User-Agents - it picks them up based on browser usage statistics from a real-world database.
Problem
I am learning Scrapy, a web crawling framework. I know I can set `USER_AGENT` in `settings.py` file of the Scrapy project. When I run the Scrapy, I can see the `USER_AGENT`'s value in `INFO` logs. This `USER_AGENT` gets set in every download request to the server I want to crawl. But I am using multiple `USER_AGENT` randomly with the help of this solution. I guess this randomly chosen `USER_AGENT` would be working. I want to confirm it. So, how I can make Scrapy shows `USER_AGENT` per download request so I can see the value of `USER_AGENT` in the logs?