Trouble converting long list of data.frames (~1 million) to single data.frame using do.call and ldply

Question

I know there are many questions here in SO about ways to convert a list of data.frames to a single data.frame using do.call or ldply, but this questions is about understanding the inner workings of both methods and trying to figure out why I can't get either to work for concatenating a list of almost 1 million df's of the same structure, same field names, etc. into a single data.frame. Each data.frame is of one row and 21 columns. The data started out as a JSON file, which I converted to lists using fromJSON, then ran another lapply to extract part of the list and converted to data.frame and ended up with a list of data.frames. I've tried: ``` df <- do.call("rbind", list) df <- ldply(list) ``` but I've had to kill the process after letting it run up to 3 hours and not getting anything back. Is there a more efficient method of doing this? How can I troubleshoot what is happening and why is it taking so long? FYI - I'm using RStudio server on a 72GB quad-core server with RHEL, so I don'…

Accepted Answer

Given that you are looking for performance, it appears that a `data.table` solution should be suggested.

There is a function `rbindlist` which is the `same` but much faster than `do.call(rbind, list)`

```
library(data.table)
X <- replicate(50000, data.table(a=rnorm(5), b=1:5), simplify=FALSE)
system.time(rbindlist.data.table <- rbindlist(X))
##  user  system elapsed 
##  0.00    0.01    0.02
```

It is also very fast for a list of `data.frame`

```
Xdf <- replicate(50000, data.frame(a=rnorm(5), b=1:5), simplify=FALSE)

system.time(rbindlist.data.frame <- rbindlist(Xdf))
##  user  system elapsed 
##  0.03    0.00    0.03
```

For comparison

```
system.time(docall <- do.call(rbind, Xdf))
##  user  system elapsed 
## 50.72    9.89   60.88 
```

And some proper benchmarking

```
library(rbenchmark)
benchmark(rbindlist.data.table = rbindlist(X), 
           rbindlist.data.frame = rbindlist(Xdf),
           docall = do.call(rbind, Xdf),
           replications = 5)
##                   test replications elapsed    relative user.self sys.self 
## 3               docall            5  276.61 3073.444445    264.08     11.4 
## 2 rbindlist.data.frame            5    0.11    1.222222      0.11      0.0 
## 1 rbindlist.data.table            5    0.09    1.000000      0.09      0.0 
```

and against @JoshuaUlrich's solutions

```
benchmark(use.rbl.dt  = rbl.dt(X), 
          use.rbl.ju  = rbl.ju (Xdf),
          use.rbindlist =rbindlist(X) ,
          replications = 5)

##              test replications elapsed relative user.self 
## 3  use.rbindlist            5    0.10      1.0      0.09
## 1     use.rbl.dt            5    0.10      1.0      0.09
## 2     use.rbl.ju            5    0.33      3.3      0.31 
```

I'm not sure you really need to use `as.data.frame`, because a `data.table` inherits class `data.frame`

Trouble converting long list of data.frames (~1 million) to single data.frame using do.call and ldply

Solution

Problem

Related problems