FAQ markup to R data structure

markdown, markup, parsing, r, r-faq

Solution

(This addresses point 3.)

You can convert the texinfo file to XML

wget http://cran.r-project.org/doc/FAQ/R-FAQ.texi
makeinfo --xml R-FAQ.texi

and then read it with the XML package.

library(XML)
doc <- xmlParse("R-FAQ.xml")
r <- xpathSApply( doc, "//node", function(u) {
  list(list(
    title    = xpathSApply(u, "nodename", xmlValue),
    contents = as(u, "character")
  ))
} )
free(doc)

But it is much easier to convert it to text

makeinfo --plaintext R-FAQ.texi > R-FAQ.txt

and parse the result manually.

doc <- readLines("R-FAQ.txt")

# Split the document into questions
# i.e., around lines like ****** or ======.
i <- grep("[*=]{5}", doc) - 1
i <- c(1,i)
j <- rep(seq_along(i)[-length(i)], diff(i))
stopifnot(length(j) == length(doc))
faq <- split(doc, j)

# Clean the result: since the questions 
# are in the subsections, we can discard the sections.
faq <- faq[ sapply(faq, function(u) length(grep("[*]", u[2])) == 0) ]

# Use the result
cat(faq[[ sample(seq_along(faq),1) ]], sep="\n")

Problem

I'm reading the R FAQ source in texinfo, and thinking that it would be easier to manage and extend if it was parsed as an R structure. There are several existing examples related to this: the fortunes package bibtex entries Rd files each with some desirable features. In my opinion, FAQs are underused in the R community because they lack i) easy access from the R command-line (ie through an R package); ii) powerful search functions; iii) cross-references; iv) extensions for contributed packages. Drawing ideas from packages `bibtex` and `fortunes`, we could conceive a new system where: FAQs can be searched from R. Typical calls would resemble the `fortune()` interface: `faq("lattice print")`, or `faq() #surprise me!`, `faq(51)`, `faq(package="ggplot2")`. Packages can provide their own `FAQ.rda`, the format of which is not clear yet (see below) `Sweave`/`knitr` drivers are provided to output nicely formatted Markdown/LaTeX, etc. QUESTION I'm not sure what is the best input format, however. Either for converting the existing FAQ, or for adding new entries. It is rather cumbersome to use R syntax with a tree of nested lists (or an ad hoc S3/S4/ref `class` or `structure`, ``` \list(title = "Something to be \\escaped", entry = "long text with quotes, links and broken characters", category = c("windows", "mac", "test")) ``` `Rd` documentation, even though not an R structure per se (it is more a subset of LaTeX with its own parser), can perhaps provide a more appealing example of an input format. It also has a set of tools to parse the structure in R. However, its current purpose is rather specific and different, being oriented towards general documentation of R functions, not FAQ entries. Its syntax is not ideal either, I think a more modern markup, something like markdown, would be more readable. Is there something else out there, maybe examples of parsing markdown files into R structures? An example of deviating Rd files away from their intended purpose? To summarise I would like to come up with: 1- a good design for an R structure (class, perhaps) that would extend the `fortune` package to more general entries such as FAQ items 2- a more convenient format to enter new FAQs (rather than the current texinfo format) 3- a parser, either written in R or some other language (bison?) to convert the existing FAQ into the new structure (1), and/or the new input format (2) into the R structure. Update 2: in the last two days of the bounty period I got two answers, both interesting but completely different. Because the question is quite vast (arguably ill-posed), none of the answers provide a complete solution, thus I will not (for now anyway) accept an answer. As for the bounty, I'll attribute it to the answer most up-voted before the bounty expires, wishing there was a way to split it more equally.

Original source