Scraping with __doPostBack with link url hidden

Question

I am trying to scrape search results from website that uses a `__doPostBack` function. The webpage displays 10 results per search query. To see more results, one has to click a button that triggers a `__doPostBack` javascript. After some research, I realized that the POST request behaves just like a form, and that one could simply use scrapy's `FormRequest` to fill that form. I used the following thread: Troubles using scrapy with javascript __doPostBack method to write the following script. ``` # -*- coding: utf-8 -*- from scrapy.contrib.spiders import CrawlSpider from scrapy.http import FormRequest from scrapy.http import Request from scrapy.selector import Selector from ahram.items import AhramItem import re class MySpider(CrawlSpider): name = u"el_ahram2" def start_requests(self): search_term = u'اقتصاد' baseUrl = u'http://digital.ahram.org.eg/sresult.aspx?srch=' + search_term + u'&archid=1' requests = [] for i in range(1, 4):#crawl first 3 pages as a test argument = u"'Page$"+ st…

Accepted Answer

Your code is fine. `fetch_articles` is running. You can test it by adding a print statement.

However, the website requires you to validate POST requests. In order to validate them, you must have `__EVENTVALIDATION` and `__VIEWSTATE` in your request body to prove you are responding to their form. In order to get these, you need to first make a GET request, and extract these fields from the form. If you don't provide this, you get an error page instead, which did not contain any links with "checkpart.aspx?Serial=", so your `for` loop was not being executed.

Here is how I've setup the `start_request`, and then `fetch_search` does what `start_request` used to do.

```
class MySpider(CrawlSpider):
    name = u"el_ahram2"

def start_requests(self):
        search_term = u'اقتصاد'
        baseUrl = u'http://digital.ahram.org.eg/sresult.aspx?srch=' + search_term + u'&archid=1'
        SearchPage = Request(baseUrl, callback = self.fetch_search)
        return [SearchPage]

def fetch_search(self, response):
        sel = Selector(response)
        search_term = u'اقتصاد'
        baseUrl = u'http://digital.ahram.org.eg/sresult.aspx?srch=' + search_term + u'&archid=1'
        viewstate = sel.xpath("//input[@id='__VIEWSTATE']/@value").extract().pop()
        eventvalidation = sel.xpath("//input[@id='__EVENTVALIDATION']/@value").extract().pop()
        for i in range(1, 4):#crawl first 3 pages as a test
            argument =  u"'Page$"+ str(i+1) + u"'"
            data = {'__EVENTTARGET': u"'GridView1'", '__EVENTARGUMENT': argument, '__VIEWSTATE': viewstate, '__EVENTVALIDATION': eventvalidation}
            currentPage = FormRequest(baseUrl, formdata = data, callback = self.fetch_articles)
            yield currentPage

...
```

Scraping with __doPostBack with link url hidden

Solution

Problem