how to aggregate 30 granular data to 5 minute data

r, time-series

Solution

Versions of this question have been asked and answered a bunch of times on stackoverflow. Yet it keeps getting asked. Here's an answer that, hopefully, meets the needs of most:

First, use a package that handles irregular time series. It makes it much easier. I like `xts`.

library(xts)

mydata <- structure(list(DATE = structure(c(1362114023, 1362114053, 1362114083, 
1362114113, 1362114143, 1362114150, 1362114173, 1362114180, 1362114203, 
1362114210, 1362114233, 1362114240, 1362114263, 1362114270, 1362114293, 
1362114300, 1362114330, 1362114360, 1362114390, 1362114420, 1362114450, 
1362114480, 1362114510, 1362114540, 1362114570, 1362114600, 1362114630, 
1362114660, 1362114690, 1362114720, 1362114750, 1362114780, 1362114810, 
1362114840, 1362114870, 1362114900, 1362114930, 1362114960, 1362114990, 
1362115020, 1362115050, 1362115080, 1362115111, 1362115141, 1362115171, 
1362115201, 1362115231, 1362115261, 1362115291, 1362115321), class = c("POSIXct", 
"POSIXt"), tzone = ""), CPU = c(30L, 29L, 28L, 29L, 27L, 10L, 
25L, 11L, 23L, 9L, 22L, 8L, 22L, 7L, 19L, 7L, 7L, 8L, 6L, 7L, 
6L, 7L, 8L, 8L, 7L, 6L, 8L, 8L, 9L, 8L, 9L, 10L, 9L, 8L, 8L, 
6L, 8L, 7L, 9L, 10L, 11L, 11L, 9L, 9L, 8L, 9L, 11L, 8L, 6L, 8L
)), .Names = c("DATE", "CPU"), row.names = c(132611L, 132612L, 
132613L, 132614L, 132615L, 131428L, 132616L, 131429L, 132617L, 
131430L, 132618L, 131431L, 132619L, 131432L, 132620L, 131433L, 
131434L, 131435L, 131436L, 131437L, 131438L, 131439L, 131440L, 
131441L, 131442L, 131443L, 131444L, 131445L, 131446L, 131447L, 
131448L, 131449L, 131450L, 131451L, 131452L, 131453L, 131454L, 
131455L, 131456L, 131457L, 131458L, 131459L, 131460L, 131461L, 
131462L, 131463L, 131464L, 131465L, 131466L, 131467L), class = "data.frame")

mydata.xts <- xts(mydata$CPU, order.by = mydata$DATE)

Then, adapt the `period.apply` infrastructure to make it easy to aggregate to different windows on-the-fly:

apply.periodly <- function (x, FUN, period, k=1, ...) 
{
  if (!require("xts")) {
    stop("Need 'xts'")
  }
  ep <- endpoints(x, on=period, k=k)
  period.apply(x, ep, FUN, ...)
}

Now, create your aggregations.

mydata.10m <- apply.periodly(x = mydata.xts, FUN = mean, period = "minutes", k = 10)
mydata.5m <- apply.periodly(x = mydata.xts, FUN = mean, period = "minutes", k = 5)

Note that the output timestamp will reflect the last input timestamp in each aggregation window.

mydata.10m
                     [,1]
2013-03-01 00:09:30 14.80
2013-03-01 00:19:31  8.55
2013-03-01 00:22:01  8.40

mydata.5m
                        [,1]
2013-03-01 00:04:53 19.93333
2013-03-01 00:09:30  7.10000
2013-03-01 00:14:30  8.30000
2013-03-01 00:19:31  8.80000
2013-03-01 00:22:01  8.40000

However, you can round your timestamps up or down:

align.time.down=function(x,n){index(x)=index(x)-n;align.time(x,n)}

mydata.10m <- align.time(mydata.10m, 10*60)
mydata.10m
#                      [,1]
# 2013-03-01 00:10:00 14.80
# 2013-03-01 00:20:00  8.55
# 2013-03-01 00:30:00  8.40

mydata.5m <- align.time.down(mydata.5m, 5*60)
mydata.5m
#                         [,1]
# 2013-03-01 00:00:00 19.93333
# 2013-03-01 00:05:00  7.10000
# 2013-03-01 00:10:00  8.30000
# 2013-03-01 00:15:00  8.80000
# 2013-03-01 00:20:00  8.40000

Problem

Original source