Pig: loading a data file using an external schema file

apache-pig, gruntjs, load, schema

Solution

The AS clause is for specifying the schema directly not the path to the schema file.

 A = LOAD '<file path>' USING PigStorage('\u0001') as 'type: long, id:chararray, nameformat:chararray';

Alternatively, a file named `.pig_schema` containing the schema and located in your input directory could work as well. Never tried that though. It must be a JSON file with the following syntax:

{"fields":[
        {"name":"type","type":55,"description":"Fu","schema":null},
        {"name":"id","type":15,"description":"Bar","schema":null},
        {"name":"nameFormat","type":55,"description":"Xu","schema":null},
    ] ,"version":0,"sortKeys":[],"sortKeyOrders":[]}

This file is also generated if you specify the -schema option when storing with PigStorage.

Problem

I have a data file and a corresponding schema file stored in separate locations. I would like to load the data using the schema in the schema-file. I tried using ``` A= LOAD '<file path>' USING PigStorage('\u0001') as '<schema-file path>' ``` but get an error. What is the syntax for correctly loading the file? The schema file format is something like: ``` data1 - complex - - - - format - - data1 event_type - - - - - long - "ends '\001'" data1 event_id - - - - - varchar(50) - "ends '\001'" data1 name_format - - - - - varchar(10) - "ends newline" ```

Original source