HtmlUnit failing when it tries to open dead javascript links. Is there a way to tell it not to load specific URLs?

Question

I'm trying to do a little scraping on this site to programatically find polling info. I originally tried Python, which worked great for loading the site and navigating around the `aspx` forms, but couldn't extract the embedded maps data (since no packages (as of yet) handle javascript). So I've opted to dust off my Java skills and break out HtmlUnit. However, I almost instantly hit a snag. It appears as though there are some dead links to javascript files on the site that don't exists. When HtmlUnit tries to load them it gets a 404 and self destructs. Specific Error ``` Jul 21, 2013 9:51:22 PM com.gargoylesoftware.htmlunit.html.HtmlPage loadExternalJavaScriptFile SEVERE: Error loading JavaScript from [http://www.eci-polldaymonitoring.nic.in/psl/GoogleMapForASPNet.ascx/jsdebug]. com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException: 404 Not Found for http://www.eci-polldaymonitoring.nic.in/psl/GoogleMapForASPNet.ascx/jsdebug at com.gargoylesoftware.htmlunit.WebClient.throwFailin…

Accepted Answer

A brief look at the HtmlUnit JavaDoc seems to indicate that you should be able to use WebClientOptions#setExceptionOnFailingStatusCode(boolean)

e.g.,

```
import com.gargoylesoftware.htmlunit.WebClient
import com.gargoylesoftware.htmlunit.BrowserVersion
import com.gargoylesoftware.htmlunit.html.HtmlPage

object ScrapeTest {

def main(args: Array[String]): Unit = {
    val pageurl = "http://www.eci-polldaymonitoring.nic.in/psl/"
    val client = new WebClient(BrowserVersion.INTERNET_EXPLORER_8)
    // Don't throw exception on failing status code
    client.getOptions.setExceptionOnFailingStatusCode(false)

var response: HtmlPage = client.getPage(pageurl)

println(response.asText())
  }
}
```

If that doesn't work you could also try:

- WebClientOptions#setJavaScriptEnabled(boolean)

- WebClientOptions#setThrowExceptionOnScriptError(boolean)

HtmlUnit failing when it tries to open dead javascript links. Is there a way to tell it not to load specific URLs?

Solution

Problem