403 Error When Using Rvest to Log Into Website For Scraping

Question

I am trying to scrape a page on a website that requires a login and am consitently getting a 403 Error. I have modified the code from these 2 posts for my site, Using rvest or httr to log in to non-standard forms on a webpage and how to reuse a session to avoid repeated login when scraping with rvest? ``` library(rvest) pgsession <- html_session("https://www.optionslam.com/earnings/stocks/MSFT?page=-1") pgform <- html_form(pgsession)[[1]] filled_form <- set_values(pgform, 'username'='user', 'password'='pass') s <- submit_form(pgsession, filled_form) # s is your logged in session ``` When the code is run, I get this message: ``` Submitting with 'NULL' Warning message: In request_POST(session, url = url, body = request$values, encode = request$encode, : Forbidden (HTTP 403). ``` I have also run the code this way, by updating user_agent as R.S. suggested in the comments, however, I receive the same error as above. ``` library(rvest) library(httr) uastring <- "Mozilla/5.0 (Macintosh; Inte…

Accepted Answer

Using R.S.'s suggestion, I used RSelenium to log in successfully.

A quick note for fellow mac users on using either chrome or phantom. I am running El Capitan so had some issue getting the mac to recognize the paths to both of the bin files. Instead, I moved the bin files to /usr/local/bin and they ran without an issue.

Below is the code to do so:

```
library(RSelenium)
RSelenium::startServer()
remDr <- remoteDriver(browserName = "chrome")
remDr$open()
appURL <- 'https://www.optionslam.com/accounts/login/'
remDr$navigate(appURL)
remDr$findElement("id", "id_username")$sendKeysToElement(list("user"))
remDr$findElement("id", "id_password")$sendKeysToElement(list("password", key='enter'))

appURL <- 'https://www.optionslam.com/earnings/stocks/MSFT?page=-1'
remDr$navigate(appURL)
```

This can also be done with phantom,

```
library(RSelenium)

pJS <- phantom() # start phantomjs

appURL <- 'https://www.optionslam.com/accounts/login/'
remDr <- remoteDriver(browserName = "phantomjs")
remDr$open()
remDr$navigate(appURL)
remDr$findElement("id", "id_username")$sendKeysToElement(list("user"))
remDr$findElement("id", "id_password")$sendKeysToElement(list("password", key='enter'))

appURL <- 'https://www.optionslam.com/earnings/stocks/MSFT?page=-1'
remDr$navigate(appURL)
```

403 Error When Using Rvest to Log Into Website For Scraping

Solution

Problem

Related problems