Confused about running Scrapy from within a Python script

Question

Following document, I can run scrapy from a Python script, but I can't get the scrapy result. This is my spider: ``` from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from items import DmozItem class DmozSpider(BaseSpider): name = "douban" allowed_domains = ["example.com"] start_urls = [ "http://www.example.com/group/xxx/discussion" ] def parse(self, response): hxs = HtmlXPathSelector(response) rows = hxs.select("//table[@class='olt']/tr/td[@class='title']/a") items = [] # print sites for row in rows: item = DmozItem() item["title"] = row.select('text()').extract()[0] item["link"] = row.select('@href').extract()[0] items.append(item) return items ``` Notice the last line, I try to use the returned parse result, if I run: ``` scrapy crawl douban ``` the terminal could print the return result But I can't get the return result from the Python script. Here is my Python script: ``` from twisted.internet import reactor from scrapy.crawler import Crawler from…

Accepted Answer

Terminal prints the result because the default log level is set to `DEBUG`.

When you are running your spider from the script and call `log.start()`, the default log level is set to `INFO`.

Just replace:

```
log.start()
```

with

```
log.start(loglevel=log.DEBUG)
```

UPD:

To get the result as string, you can log everything to a file and then read from it, e.g.:

```
log.start(logfile="results.log", loglevel=log.DEBUG, crawler=crawler, logstdout=False)

reactor.run()

with open("results.log", "r") as f:
    result = f.read()
print result
```

Hope that helps.

Confused about running Scrapy from within a Python script

Solution

Problem

Related problems