Understanding exactly when a data.table is a reference to (vs a copy of) another data.table
assignment-operator, copy, data.table, r, reference
Solution
Yes, it's subassignment in R using `<-` (or `=` or `->`) that makes a copy of the whole object. You can trace that using `tracemem(DT)` and `.Internal(inspect(DT))`, as below. The `data.table` features `:=` and `set()` assign by reference to whatever object they are passed. So if that object was previously copied (by a subassigning `<-` or an explicit `copy(DT)`) then it's the copy that gets modified by reference.
DT <- data.table(a = c(1, 2), b = c(11, 12))
newDT <- DT
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
.Internal(inspect(newDT)) # precisely the same object at this point
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
tracemem(newDT)
# [1] "<0x0000000003b7e2a0"
newDT$b[2] <- 200
# tracemem[0000000003B7E2A0 -> 00000000040ED948]:
# tracemem[00000000040ED948 -> 00000000040ED830]: .Call copy $<-.data.table $<-
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),TR,ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,12
# ATTRIB: # ..snip..
.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,200
# ATTRIB: # ..snip..
Notice how even the `a` vector was copied (different hex value indicates new copy of vector), even though `a` wasn't changed. Even the whole of `b` was copied, rather than just changing the elements that need to be changed. That's important to avoid for large data, and why `:=` and `set()` were introduced to `data.table`.
Now, with our copied `newDT` we can modify it by reference :
newDT
# a b
# [1,] 1 11
# [2,] 2 200
newDT[2, b := 400]
# a b # See FAQ 2.21 for why this prints newDT
# [1,] 1 11
# [2,] 2 400
.Internal(inspect(newDT))
# @0000000003D97A58 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040ED7F8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040ED8D8 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,400
# ATTRIB: # ..snip ..
Notice that all 3 hex values (the vector of column points, and each of the 2 columns) remain unchanged. So it was truly modified by reference with no copies at all.
Or, we can modify the original `DT` by reference :
DT[2, b := 600]
# a b
# [1,] 1 11
# [2,] 2 600
.Internal(inspect(DT))
# @0000000003B7E2A0 19 VECSXP g0c7 [OBJ,NAM(2),ATT] (len=2, tl=100)
# @00000000040C2288 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 1,2
# @00000000040C2250 14 REALSXP g0c2 [NAM(2)] (len=2, tl=0) 11,600
# ATTRIB: # ..snip..
Those hex values are the same as the original values we saw for `DT` above. Type `example(copy)` for more examples using `tracemem` and comparison to `data.frame`.
Btw, if you `tracemem(DT)` then `DT[2,b:=600]` you'll see one copy reported. That is a copy of the first 10 rows that the `print` method does. When wrapped with `invisible()` or when called within a function or script, the `print` method isn't called.
All this applies inside functions too; i.e., `:=` and `set()` do not copy on write, even within functions. If you need to modify a local copy, then call `x=copy(x)` at the start of the function. But, remember `data.table` is for large data (as well as faster programming advantages for small data). We deliberately don't want to copy large objects (ever). As a result we don't need to allow for the usual 3* working memory factor rule of thumb. We try to only need working memory as large as one column (i.e. a working memory factor of 1/ncol rather than 3).
Problem
I'm having a little trouble understanding the pass-by-reference properties of `data.table`. Some operations seem to 'break' the reference, and I'd like to understand exactly what's happening. On creating a `data.table` from another `data.table` (via `<-`, then updating the new table by `:=`, the original table is also altered. This is expected, as per: `?data.table::copy` and stackoverflow: pass-by-reference-the-operator-in-the-data-table-package Here's an example: ``` library(data.table) DT <- data.table(a=c(1,2), b=c(11,12)) print(DT) # a b # [1,] 1 11 # [2,] 2 12 newDT <- DT # reference, not copy newDT[1, a := 100] # modify new DT print(DT) # DT is modified too. # a b # [1,] 100 11 # [2,] 2 12 ``` However, if I insert a non-`:=` based modification between the `<-` assignment and the `:=` lines above, `DT` is now no longer modified: ``` DT = data.table(a=c(1,2), b=c(11,12)) newDT <- DT newDT$b[2] <- 200 # new operation newDT[1, a := 100] print(DT) # a b # [1,] 1 11 # [2,] 2 12 ``` So it seems that the `newDT$b[2] <- 200` line somehow 'breaks' the reference. I'd guess that this invokes a copy somehow, but I would like to understand fully how R is treating these operations, to ensure I don't introduce potential bugs in my code. I'd very much appreciate if someone could explain this to me.