How can i use scrapy shell to with parameters on url

django, python, scrapy

Solution

apparently, you need to enclose your url within double quotes:

scrapy shell "http://www.seek.com.au/JobSearch?DateRange=31&SearchFrom=quick&Keywords=python&nation=3000"
>>> from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
>>> lx = SgmlLinkExtractor()

then everything works smoothly (this above is my actual shell output)

tried it without double quotes, doesn't work (the fetch thread keeps running and first key press exits to bash without changing my visual output, thus giving me the same error you have)

Problem

I want to scrap the job website. i want to do some testing in scrapy shell. Hence if i type this `scrapy shell http://www.seek.com.au` Then if i type `from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor` then it works fine But if i do this ``` scrapy shell http://www.seek.com.au/JobSearch?DateRange=31&SearchFrom=quick&Keywords=python&nation=3000 ``` Then if i type `from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor` Then it says invalid bash command `from` and it exits the scrapy job and come on screen as stopped job ``` >>> from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor -bash: from: command not found [5]+ Stopped scrapy shell http://www.seek.com.au/JobSearch?DateRange=31 [7] Done Keywords=php ```

Original source