How to use PyCharm to debug Scrapy projects
debugging, pycharm, python, python-2.7, scrapy
Solution
The `scrapy` command is a python script which means you can start it from inside PyCharm.
When you examine the scrapy binary (`which scrapy`) you will notice that this is actually a python script:
#!/usr/bin/python
from scrapy.cmdline import execute
execute()
This means that a command like `scrapy crawl IcecatCrawler` can also be executed like this: `python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler`
Try to find the scrapy.cmdline package. In my case the location was here: `/Library/Python/2.7/site-packages/scrapy/cmdline.py`
Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case `crawl IcecatCrawler`.
Like this:
Put your breakpoints anywhere in your crawling code and it should work™.
Problem
I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please? What I have tried Actually I tried to run the spider as a script. As a result, I built that script. Then, I tried to add my Scrapy project to PyCharm as a model like this: ``` File->Setting->Project structure->Add content root. ``` But I don't know what else I have to do