Scrapy Very Basic Example
python, scrapy, web-scraping
Solution
You may have better luck looking through the tutorial first, as opposed to the "Scrapy at a glance" webpage.
The tutorial implies that Scrapy is, in fact, a separate program.
Running the command `scrapy startproject tutorial` will create a folder called `tutorial` several files already set up for you.
For example, in my case, the modules/packages `items`, `pipelines`, `settings` and `spiders` have been added to the root package `tutorial` .
tutorial/
scrapy.cfg
tutorial/
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
...
The `TorrentItem` class would be placed inside `items.py`, and the `MininovaSpider` class would go inside the `spiders` folder.
Once the project is set up, the command-line parameters for Scrapy appear to be fairly straightforward. They take the form:
scrapy crawl <website-name> -o <output-file> -t <output-type>
Alternatively, if you want to run scrapy without the overhead of creating a project directory, you can use the runspider command:
scrapy runspider my_spider.py
Problem
Hi I have Python Scrapy installed on my mac and I was trying to follow the very first example on their web. They were trying to run the command: ``` scrapy crawl mininova.org -o scraped_data.json -t json ``` I don't quite understand what does this mean? looks like scrapy turns out to be a separate program. And I don't think they have a command called crawl. In the example, they have a paragraph of code, which is the definition of the class MininovaSpider and the TorrentItem. I don't know where these two classes should go to, go to the same file and what is the name of this python file?