r - how to select a different number of observation within each group

data-management, r

Solution

If you are looking for efficiency, might I suggest looking into the `data.table` package. A fairly straight-forward solution to this problem could be:

#Convert objects to data.table
require("data.table")
ToSelect <- data.table(ToSelect)
NumObs <- data.table(NumObs)

#Merge data
ToSelect <- merge(ToSelect,NumObs,by=c("key1","key2"),all.x=T)

#Provide intra-group ordering variable
ToSelect[,Grp.Seq:=seq(1:.N),by=c("key1","key2")]
Selected <- ToSelect[NumObs>=Grp.Seq]
Selected

   key1 key2 var1 NumObs Grp.Seq
1:    1    a    2      1       1
2:    1    b    4      2       1
3:    1    b    6      2       2
4:    2    a    7      2       1
5:    2    a    8      2       2
6:    2    b    1      1       1

If you are new to `R` anyway, and you often work with large datasets, it might make sense to learn `data.table` from the beginning. I work with very large datasets for my work, and the data.frame class is not really practical for much of the work I do. It is very easy to switch back between `data.frame` and `data.table` if need be.

Problem

I am relatively new to r (coming from sas) I need to select a different number of observations within each group. Groups are identified by the values of two variables ``` ToSelect <- data.frame( key1=c(1,1,1,1,1,2,2,2,2,2,2,2), key2=c("a","a","b","b","b","a","a","a","a","b","b","b"), var1=c(2,3,4,6,2,7,8,5,7,1,8,5) ) NumObs <- data.frame( key1=c(1,1,2,2), key2=c("a","b","a","b"), NumObs=c(1,2,2,1) ) ``` I tried (from question "Select first 80 observations for each level in R") ``` ToSelect <- merge(x=ToSelect,y=NumObs,by=c("key1","key2")) library(plyr) Selected <- ddply(ToSelect, .(key1,key2), head, n = NumObs) ``` which gives Error: length(n) == 1L is not TRUE which is probably an obvious error to experts (n a scalar, NumObs a vector?) From the same question, I tried: ``` Selected <- do.call( rbind, lapply(split(ToSelect, c(ToSelect$key1,ToSelect$key2)), head, NumObs) ) ``` which gives Error: length(n) == 1L is not TRUE. In addition: Warning message: In split.default(x = seq_len(nrow(x)), f = f, drop = drop, ...) : data length is not a multiple of split variable So, same error as before, plus the multiple stuff, I cannot use split if groups have different length? Then I found the question "Observation number by group" and I couldn't make the rle/sequence answer work in my case but adapting the ddply answer: ``` ToSelect <- ddply(ToSelect, .(key1, key2), function(z){ cbind(var1=z$var1,NumObs=z$NumObs, data.frame( SeqNum = seq_along(z$key2) ) ) } ) Selected <- ToSelect[ToSelect$SeqNum<=ToSelect$NumObs,c("key1","key2","var1")] ``` which works. Obviously my real data is much bigger, so is there an alternative and better way? Thanks!

Original source