STORE output to a single CSV?

apache-pig

Solution

You can do this in a few ways:

To set the number of reducers for all Pig opeations, you can use the `default_parallel` property - but this means every single step will use a single reducer, decreasing throughput:

`set default_parallel 1;`

Prior to calling STORE, if one of the operations execute is (COGROUP, CROSS, DISTINCT, GROUP, JOIN (inner), JOIN (outer), and ORDER BY), then you can use the `PARALLEL 1` keyword to denote the use of a single reducer to complete that command:

`GROUP a BY grp PARALLEL 1;`

See Pig Cookbook - Parallel Features for more information

Problem

Currently, when I STORE into HDFS, it creates many part files. Is there any way to store out to a single CSV file?

Original source