scrapy how spider returns value to another spider

python, python-2.7, scrapy

Solution

First of all, I want to thank @warwaruk, @Robin for helping me in this issue.

And the best thanks to my great teacher @pault

I found the solution and here is the algorithm:

- start scraping in the main page.

- extracting all the players' links.

- call back on each player's link to extract his information. and the request's meta includes: the number of players in the current main page and the position of the player that I want to scrap.

In the callback for each player:

4.1 extract player's information.

4.2 check if the date in the rage, if no: do nothing, if yes: check if this is the last play in the main player list. if yes, callback to the second main page.

simple code

def parse(self, response):
    currentPlayer = 0
    for each player in Players:
        currentPlayer +=1
        yield Request(player.link, meta={'currentPlayer':currentPlayer, 'numberOfPlayers':len(Players),callback = self.parsePlayer)

def parsePlayer(self,response):
    currentPlayer = meta['currentPlayer]
    numberOfPlayers = meta['numberOfPlayers']
    extract player's information
    if player[date] in range:
        if currentPlayer == numberOfPlayers:
            yield(linkToNextMainPage, callback = self.parse)
            yield playerInformatoin #in order to be written in JSON file
        else:
            yield playerInformaton

It works perfectly :)

Problem

The website that I am crawling contains many players and when I click on any player, I can go the his page. The website structure is like this: ``` <main page> <link to player 1> <link to player 2> <link to player 3> .. .. .. <link to payer n> </main page> ``` And when I click on any link, I go to player's page which is like this: ``` <player name> <player team> <player age> <player salary> <player date> ``` I want to scrap all the players those age is between 20 and 25 years. what I am doing scraping the main page using first spider. getting links using first spider. crawl each link using second spider. get the player informatoin using second spider. save this information in json file using pipeline. my question how can I return the `date` value from `second spider` to the `first spider` what i have tried I build my own middelware and i override the `process_spider_output`. it allows me to print the request but I don't know what else should I do in order to return that `date` value to my first spider any help is appreciated Edit Here is some of the code: ``` def parse(self, response): sel = Selector(response) Container = sel.css('div[MyDiv]') for player in Container: extract LINK and TITLE yield Request(LINK, meta={'Title': Title}, callback = self.parsePlayer) def parsePlayer(self,response): player = new PlayerItem(); extract DATE return player ``` I gave you the general code, not the very specific details in order to make it easy for you

Original source

Related problems