Scrapy: FormRequest doesn't auto-populate ASP.net hidden fields

Question

I'm using Scrapy to scrape some gold that's behind an authentication screen. The website uses `ASP.net` and ASP's got some stupid hidden fields littered all over the form (like `__VIEWSTATE`, `__EVENTTARGET`). When I call `FormRequest.from_response(response,...` I'm expecting that it reads these hidden fields automatically from the response and populates them in the `formdata` dictionary - which is what Scrapy's FormRequest documentation says it should do. But if that's the case, then why does the login process only work when I explicitly list these fields and populate them? ``` class ItsyBitsy(Spider): name = "itsybitsy" allowed_domains = ["website.com"] start_urls = ["http://website.com/cpanel/Default.aspx"] def parse(self, response): # Performs authentication to get past the login form sel = Selector(response) return [FormRequest.from_response(response, formdata={ 'tb_Username':'admin', 'tb_Password':'password', # The following fields should be auto populated, right? # So why does…

Accepted Answer

According to the source code, `Scrapy` uses the following CSS selector to parse the inputs out of the form:

```
descendant::textarea|descendant::select|descendant::input[@type!="submit" and @type!="image" and @type!="reset"and ((@type!="checkbox" and @type!="radio") or @checked)]
```

In other words, all of your hidden inputs are successfully parsed (and sent with the request later) with the values equal to `value` attributes. So, Scrapy does what it should here.

The login using `from_response()` doesn't work because `__EVENTTARGET` has a empty `value` attribute. If you make the login using a real browser, `__EVENTTARGET` parameter value would be set to `b_Login` via javascript `__doPostBack()` function call. And, since Scrapy cannot handle javascript (cannot call js functions), `__EVENTTARGET` is sent with an empty value which causes login failure.

`__EVENTARGUMENT` has an empty `value` too, but it is actually set to the empty string in the `__doPostBack()` function, so it doesn't make a difference here.

Hope that helps.

Scrapy: FormRequest doesn't auto-populate ASP.net hidden fields

Solution

Problem