Load local csv file to hive parquet table directly,not resort to a temp textfile table
hive, local, parquet
Solution
As stated in Hive documentation:
NO verification of data against the schema is performed by the load command.
If the file is in hdfs, it is moved into the Hive-controlled file system namespace.
You could skip a step by using `CREATE TABLE AS SELECT` for the parquet table.
So you'll have 3 steps:
- Create text table defining the schema
- Load data into text table (move the file into the new table)
- `CREATE TABLE parquet_table AS SELECT * FROM textfile_table STORED AS PARQUET;` supported from hive 0.13
Problem
I am now preparing to store data in .csv files into hive. Of course, because of the good performance of parquet file format, the hive table should is parquet format. So, the normal way, is to create a temp table whose format is `textfile`, then I load local CSV file data into this temp table, and finally, create a same-structure parquet table and use sql `insert into parquet_table values (select * from textfile_table);`. But I don't think this temp textfile table is necessary. So, my question is, is there a way for me to load these local .csv files into hive parquet-format table directly, namely, not to resort the a temp table? Or a easier way to accomplish this task?