Drop single column in Pig
apache-pig, hadoop, mapreduce
Solution
Maybe similiar-ish to this question:
- How to "update" a column using pig latin
That references a JIRA ticket: https://issues.apache.org/jira/browse/PIG-1693 which examples how you can use the .. notation to denote all the remaining fields:
D = FOREACH C GENERATE $1 .. ;
This assumes you have 0.9.0+ PIG
Problem
I'm filtering a table by a list of about 20 IDs. Right now my code looks like this: ``` A = LOAD 'ids.txt' USING PigStorage(); B = LOAD 'massive_table' USING PigStorage(); C = JOIN A BY $0, B BY $0; D = FOREACH C GENERATE $1, $2, $3, $4, ... STORE D INTO 'foo' USING PigStorage(); ``` What I don't like is line D, where I have to regenerate a new table to get rid of the joining column by explicitly declaring every single other column I want present (and sometimes that is a lot of columns). I'm wondering if there's something equivalent to: ``` FILTER B BY $0 IN (A) ``` or: ``` DROP $0 FROM C ```