Save storage space for small integers or factors with few levels

Question

R seems to require four bytes of storage per integer, even for small ones: ``` > object.size(rep(1L, 10000)) 40040 bytes ``` And, what is more, even for factors: ``` > object.size(factor(rep(1L, 10000))) 40456 bytes ``` I think, especially in the latter case this could be handled much better. Is there a solution that would help me reduce the storage requirements for this case to eight or even two bits per row? Perhaps a solution that uses the `raw` type internally for storage but behaves like a normal factor otherwise. The `bit` package offers this for bits, but I haven't found anything similar for factors. My data frame with just a few millions of rows is consuming gigabytes, and that's a huge waste of memory and run time (!). Compression will reduce the required disk space, but again at the expense of run time. Related: - Why do logicals (booleans) in R require 4 bytes? - How can I efficiently construct a very long factor with few levels?

Accepted Answer

Since you mention `raw` (and assuming you have less than 256 factor levels) - you could do the prerequisite conversion operations if memory is your bottleneck and CPU time isn't. For example:

```
f = factor(rep(1L, 1e5))
object.size(f)
# 400456 bytes

f.raw = as.raw(f)
object.size(f.raw)
#100040 bytes

# to go back:
identical(as.factor(as.integer(f.raw)), f)
#[1] TRUE
```

You can also save the factor levels separately and recover them if that's something you're interested in doing, but as far as grouping and all that goes you can just do it all with `raw` and never go back to factors (except for presentation).

If you have specific use cases where you have trouble with this method, please post it, otherwise I think this should work just fine.

Here's a starting point for your `byte.factor` class:

```
byte.factor = function(f) {
  res = as.raw(f)
  attr(res, "levels") <- levels(f)
  attr(res, "class") <- "byte.factor"
  res
}

as.factor.byte.factor = function(b) {
  factor(attributes(b)$levels[as.integer(b)], attributes(b)$levels)
}
```

So you can do things like:

```
f = factor(c('a','b'), letters)
f
#[1] a b
#Levels: a b c d e f g h i j k l m n o p q r s t u v w x y z

b = byte.factor(f)
b
#[1] 01 02
#attr(,"levels")
# [1] "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s"
#[20] "t" "u" "v" "w" "x" "y" "z"
#attr(,"class")
#[1] "byte.factor"

as.factor.byte.factor(b)
#[1] a b
#Levels: a b c d e f g h i j k l m n o p q r s t u v w x y z
```

Check out how `data.table` overrides `rbind.data.frame` if you want to make `as.factor` generic and just add whatever functions you want to add. Should all be quite straightforward.

Save storage space for small integers or factors with few levels

Solution

Problem

Related problems