scrapy allow all subdomains

python, scrapy

Solution

You can set an `allow_domains` list for the rule :

rules = (
       Rule(SgmlLinkExtractor(allow_domains=('domain1','domain2' ), ),)

For example:

rules = (
       Rule(SgmlLinkExtractor(allow_domains=('example.com','example1.com' ), ),)

This will filter allow urls like :

www.example.com/blaa/bla/
www.example1.com/blaa/bla/
www.something.example.com/blaa/bla/

Problem

I want to use Scrapy to crawl a website that it's pages are divided into a lot of subdomains I know I need a `CrawlSpider` with a `Rule` but I need the Rule to be just "allow all subdomains and let the parsers handle themselves according to the data" (meaning - in the example the item_links are in different subdomains) example for the code: ``` def parse_page(self, response): sel = Selector(response) item_links = sel.xpath("XXXXXXXXX").extract() for item_link in item_links: item_request = Request(url=item_link, callback=self.parse_item) yield item_request def parse_item(self, response): sel = Selector(response) ``` ** EDIT ** Just to make the question clear, I want the ability to crawl all of *.example.com -> meaning not to get `Filtered offsite request to 'foo.example.com'` ** ANOTHER EDIT ** Following @agstudy's answer, make sure you don't forget to delete `allowed_domains = ["www.example.com"]`

Original source