Conditional Filter in GROUP BY in Pig

apache-pig, hadoop, hadoop-streaming

Solution

I did with my way as specified in the comment above. Here is how I did it.

A = LOAD 'group.txt' USING PigStorage(',') AS (f1:int, f2:chararray, f3:int, f4:int);
G = GROUP A BY (f1, f2);

CNT = FOREACH G GENERATE group, COUNT($1) AS cnt, $1;

SPLIT CNT INTO
    CNT1 IF (cnt > 1),
    CNT2 IF (cnt == 1);


M1 = FOREACH CNT1 {
        row = FILTER $2 BY (f3 == 2);
        GENERATE FLATTEN(row);
     };

M2 = FOREACH CNT2 GENERATE FLATTEN($2);

O = UNION M1, M2;

DUMP O;

(2,N1,1,10)
(4,N2,1,10)
(1,N1,2,15)
(3,N1,2,15)
(5,N3,2,20)

Problem

I have the following dataset in which I need to merge multiple rows into one if they have the same key. At the same time, I need to pick among the multiple tuples which gets grouped. ``` 1 N1 1 10 1 N1 2 15 2 N1 1 10 3 N1 1 10 3 N1 2 15 4 N2 1 10 5 N3 1 10 5 N3 2 20 ``` For example ``` A = LOAD 'data.txt' AS (f1:int, f2:chararray, f3:int, f4:int); G = GROUP A BY (f1, f2); DUMP G; ((1,N1),{(1,N1,1,10),(1,N1,2,15)}) ((2,N1),{(2,N1,1,10)}) ((3,N1),{(3,N1,1,10),(3,N1,2,15)}) ((4,N2),{(4,N2,1,10)}) ((5,N3),{(5,N3,1,10),(5,N3,2,20)}) ``` Now, I want to pick if there are multiple tuples in collected bag, I want to filter only those which have f3==2. Here is the final data which I want: ``` ((1,N1),{(1,N1,2,15)}) -- f3==2, f3==1 is removed from this set ((2,N1),{(2,N1,1,10)}) ((3,N1),{(3,N1,2,15)}) -- f3==2, f3==1 is removed from this bag ((4,N2),{(4,N2,1,10)}) ((5,N3),{(5,N3,2,10)}) ``` Any idea how to achieve this?

Original source