How write code to web crawling and scraping in R

r, screen-scraping, web, web-crawler

Solution

This seems to work.

library(XML)
library(httr)
url <- "http://www.wikiart.org/en/claude-monet/mode/all-paintings-by-alphabet/"
hrefs <- list()
for (i in 1:23) {
  response <- GET(paste0(url,i))
  doc      <- content(response,type="text/html")
  hrefs    <- c(hrefs,doc["//p[@class='pb5']/a/@href"])
}
url      <- "http://www.wikiart.org"
xPath    <- c(pictureName = "//h1[@itemprop='name']",
              date        = "//span[@itemprop='dateCreated']",
              author      = "//a[@itemprop='author']",
              style       = "//span[@itemprop='style']",
              genre       = "//span[@itemprop='genre']")
get.picture <- function(href) {
  response <- GET(paste0(url,href))
  doc      <- content(response,type="text/html")
  info     <- sapply(xPath,function(xp)ifelse(length(doc[xp])==0,NA,xmlValue(doc[xp][[1]])))
}
pictures <- do.call(rbind,lapply(hrefs,get.picture))
head(pictures)
#      pictureName                           date     author         style           genre           
# [1,] "A Corner of the Garden at Montgeron" "1877"   "Claude Monet" "Impressionism" "landscape"     
# [2,] "A Corner of the Studio"              "1861"   "Claude Monet" "Realism"       "self-portrait" 
# [3,] "A Farmyard in Normandy"              "c.1863" "Claude Monet" "Realism"       "landscape"     
# [4,] "A Windmill near Zaandam"             NA       "Claude Monet" "Impressionism" "landscape"     
# [5,] "A Woman Reading"                     "1872"   "Claude Monet" "Impressionism" "genre painting"
# [6,] "Adolphe Monet Reading in the Garden" "1866"   "Claude Monet" "Impressionism" "genre painting"

You were actually pretty close. Your xPath is fine; one problem is that not all of the pictures have all of the information (e.g., for some of the pages the nodeSets your are trying to access are empty) - note the date for "A Windnill nead Zaandam". So the code has to deal with this possibility.

So in this example, the first loop grabs the values of the href attribute of the anchor tags for each page (1:23) and combines these into a vector of length ~1300.

To process each of these 1300 pages, and since we have to deal with missing tags, it's more straightforward to create a vector containing the xPath strings and apply that element-wise to each page. That's what the function `get.picture(...)` does. The last statement calls this function with each of the 1300 hrefs, and binds the result together row-wise, using `do.call(rbind,...)`.

Note also that this code uses the somewhat more compact indexing feature for objects of class HTMLInternalDocument: `doc[xpath]` where `xpath` is an xPath string. This avoids the use of `xpathSApply(...)`, although the latter would have worked.

Problem

Original source