RSelenium and Javascript

javascript, r, selenium, web-scraping

Solution

You dont need to use `executeScript` here:

require(RSelenium)
checkForServer()
startServer()
remDr<-remoteDriver()
remDr$open()
remDr$getStatus()
remDr$navigate("http://fyed.elections.on.ca/fyed/en/form_page_en.jsp")

p.codes<-c('m1p4v4', 'n3t2y3', 'n2h3v1')
webElem<-remDr$findElement(using = 'id', value = "pcode")
webElem$sendKeysToElement(list(p.codes[1])) # send the first post code to the element

remDr$findElement("id", "en_btn_arrow")$clickElement() # find the submit button and click it

if you wanted to use `executeScript` instead you would replace the last line with:

remDr$executeScript("arguments[0].click();"
                , list(remDr$findElement("id", "en_btn_arrow")))

`executeScript` takes a script as an argument and a list. If any elements of the list are of class `webElement` then they can be referred to in the script like a DOM element. In this case the first element (zero index in JavaScript) is a `webElement` and we ask to click it in our JavaScript.

Furthermore if you examine the source code behind the button you will find when it is pressed it calls `document.pcode.submit()` so more simply in this case if you wanted to use `executeScript` you could do:

remDr$executeScript("document.pcode.submit();")

Problem

I'm fairly proficient in R, but completely ignorant regarding javaScript and other languages. I would like to like to access information on this publicly-available data set (http://fyed.elections.on.ca/fyed/en/form_page_en.jsp). In particular, I have a list of several thousand postal codes of the form ('A1A1A1') in a data frame. I would like to submit each of these postal codes to this website and then extract the name of the electoral district that is returned. RSelenium seems ideal, but I cannot figure out how to get the javascript to work. I am working on a Mac OS 10.9.5, with R 3.0.3 and RSelenium_1.3. Firefox is v. 33 and Selenium is 2.44. The following script works. ``` require(RSelenium) checkForServer() startServer() remDr<-remoteDriver() remDr$open() remDr$getStatus() remDr$navigate("http://fyed.elections.on.ca/fyed/en/form_page_en.jsp") #After inspecting the source code, you can see the input box has the id 'pcode', for postal code webElem<-remDr$findElement(using = 'id', value = "pcode") webElem$getElementAttribute('id') #This is where I am stuck remDr$executeScript(script='arguments[0].click(m1p4v4)', list(webElem)) #Utlimately, I have a list of several thousand postal codes, so I would like to create a loop through to extract all the district names that are stored on the pages that are returned with a successful javascript (see previous command). Three real postal codes that return results are as follows: p.codes<-c('m1p4v4', 'n3t2y3', 'n2h3v1') ``` I feel like I just don't understand the javascript commands necessary or the syntax of executeScript to make this work. I'd appreciate any help.

Original source