Why this is so slow? (loop in a DF row vs. a standalone vector)

Question

I have a piece of code and total elapsed time is around 30 secs of which, the following code is around 27 secs. I narrowed the offending code to this: ``` d$dis300[i] <- h ``` So I change to this other piece and is now working really fast (as expected). My question is why this is too slow against the second. The datos DF is around 7500x18 vars First: (27 sec elapsed) ``` d$dis300 <- 0 for (i in 1:netot) { h <- aaa[d$ent[i], d$dis[i]] if (h == 0) writeLines(sprintf("ERROR. ent:%i dis:%i", d$ent[i], d$dis[i])) d$dis300[i] <- h } ``` Second: (0.2 sec elapsed) ``` d$dis300 <- 0 for (i in 1:netot) { h <- aaa[d$ent[i], d$dis[i]] if (h == 0) writeLines(sprintf("ERROR. ent:%i dis:%i", d$ent[i], d$dis[i])) foo[i] <- h } d$foo <- foo ``` You can see both are the "same" but the offending one has this DF instead of a single vector. Any comment is really appreciated. I came from another type of languages and this drove me nuts for a while. At least I have solution but I like to prevent this kind o…

Accepted Answer

The reason is that `d$dis300[i] <- h` calls `$<-.data.frame`.

It's a rather complex function as you can see:

```
`$<-.data.frame`
```

You don't say what `foo` is, but if it is an atomic vector, the `$<-` function is implemented in C for speed.

Still, I hope you declare foo as follows:

```
foo <- numeric(netot)
```

This will ensure you don't need to reallocate the vector for each assignment in the loop:

```
foo <- 0 # BAD!
system.time( for(i in 1:5e4) foo[i] <- 0 ) # 4.40 secs
foo <- numeric(5e4) # Pre-allocate
system.time( for(i in 1:5e4) foo[i] <- 0 ) # 0.09 secs
```

Using the `*apply` family instead you don't worry about that:

```
d$foo <- vapply(1:netot, function(i, aaa, ent, dis) {
  h <- aaa[ent[i], dis[i]]
  if (h == 0) writeLines(sprintf("ERROR. ent:%i dis:%i", ent[i], dis[i]))
  h
}, numeric(1), aaa=aaa, ent=d$ent, dis=d$dis)
```

...here I also extracted `d$ent` and `d$dis` outside the loop which should improve things a bit too. Can't run it myself though since you didn't give reproducible data. But here's a similar example:

```
d <- data.frame(x=1)
system.time( vapply(1:1e6, function(i) d$x, numeric(1)) )         # 3.20 secs
system.time( vapply(1:1e6, function(i, x) x, numeric(1), x=d$x) ) # 0.56 secs
```

... but finally it seems it can all be reduced to (barring your error detection code):

```
d$foo <- aaa[cbind(d$ent, d$dis)]
```

Why this is so slow? (loop in a DF row vs. a standalone vector)

Solution

Problem

Related problems