R 'aggregate' runs out of memory

aggregate, memory, r

Solution

Use package data.table:

mydata <- read.table(text="       uid              mid    year month date hour min sec
1738914174 3342412291119279 2011     8    3   21   4  12
1738914174 3342413045470746 2011     8    3   21   7  12
1738914174 3342823219232783 2011     8    5    0  17   5
1738914174 3343095924467484 2011     8    5   18  20  43
1738914174 3343131303394795 2011     8    5   20  41  18
1738914174 3343386263030889 2011     8    6   13  34  25", 
header=TRUE, colClasses = c(rep("character",2),rep("numeric",6)), 
stringsAsFactors = FALSE)

library(data.table)
DT <- data.table(mydata)
DT[, length(unique(na.omit(mid))), by=list(uid,hour)]

`aggregate` coerces the grouping variables to factors, which is probably eating your memory (I assume that you have many levels of `uid`).

There might be more potential for optimization, but you don't provide a representative test case.

Problem

I've got a dataset (600 Mb wiht 5038720 observations) about micro blogging and I tried to figure out how many tweets (tweets with the same mid count as one) one users posted in an hour. Here is how the dataset looks like: ``` head(mydata) uid mid year month date hour min sec 1738914174 3342412291119279 2011 8 3 21 4 12 1738914174 3342413045470746 2011 8 3 21 7 12 1738914174 3342823219232783 2011 8 5 0 17 5 1738914174 3343095924467484 2011 8 5 18 20 43 1738914174 3343131303394795 2011 8 5 20 41 18 1738914174 3343386263030889 2011 8 6 13 34 25 ``` and here is my code: ``` count <- function(x) { length(unique(na.omit(x))) } attach(mydata) hourPost <- aggregate(mid, by=list(uid, hour), FUN=count) ``` It hanged there for about half an hour and I found that all the real memory (24 Gb) was used and it started to use the virtual memory. Any idea why this little task consumed so much time and memory and how should I improve it? Thanks in advance!

Original source