take top N after groupBy and treat them as RDD

apache-spark, rdd, scala

Solution

Your question is a little confusing, but I think this does what you're looking for:

val flattenedTopNPerGroup = 
    topNPerGroup.flatMap({case (key, numbers) => numbers.map(key -> _)})

and in the repl it prints out what you want:

flattenedTopNPerGroup.collect.foreach(println)
(foo,3)
(foo,2)
(bar,6)
(bar,5)

Problem

I'd like to get top `N` items after groupByKey of `RDD` and convert the type of `topNPerGroup`(in the below) to `RDD[(String, Int)]` where `List[Int]` values are `flatten` The `data` is ``` val data = sc.parallelize(Seq("foo"->3, "foo"->1, "foo"->2, "bar"->6, "bar"->5, "bar"->4)) ``` The top `N` items per group are computed as: ``` val topNPerGroup: RDD[(String, List[Int]) = data.groupByKey.map { case (key, numbers) => key -> numbers.toList.sortBy(-_).take(2) } ``` The result is ``` (bar,List(6, 5)) (foo,List(3, 2)) ``` which was printed by ``` topNPerGroup.collect.foreach(println) ``` If I achieve, `topNPerGroup.collect.foreach(println)` will generate (expected result!) ``` (bar, 6) (bar, 5) (foo, 3) (foo, 2) ```

Original source