Problems with Hadoop distcp from HDFS to Amazon S3
amazon-s3, amazon-web-services, hadoop
Solution
You should use s3n instead of s3.
s3n is the native file system implementation (ie - regular files), using s3 imposes hdfs block structure on the files so you can't really read them without going through hdfs libraries.
Thus:
hadoop distcp hdfs://file/1 s3n://bucket/destination
Problem
I am trying to move data from HDFS to S3 using `distcp`. The `distcp` job seems to succeed, but on S3 the files are not being created correctly. There are two issues: - The file names and paths are not replicated. All files end up as `block_<some number>` at the root of the bucket. - It creates bunch of extra files on S3 with some meta data and logs. I could not find any documentation/examples for this. What am I missing? How can I debug? Here are some more details: ``` $ hadoop version Hadoop 0.20.2-cdh3u0 Subversion -r Compiled by diego on Sun May 1 15:42:11 PDT 2011 From source with checksum hadoop fs –ls hdfs://hadoopmaster/data/paramesh/ …<bunch of files>… hadoop distcp hdfs://hadoopmaster/data/paramesh/ s3://<id>:<key>@paramesh-test/ $ ./s3cmd-1.1.0-beta3/s3cmd ls s3://paramesh-test DIR s3://paramesh-test// DIR s3://paramesh-test/test/ 2012-05-10 02:20 0 s3://paramesh-test/block_-1067032400066050484 2012-05-10 02:20 8953 s3://paramesh-test/block_-183772151151054731 2012-05-10 02:20 11209 s3://paramesh-test/block_-2049242382445148749 2012-05-10 01:40 1916 s3://paramesh-test/block_-5404926129840434651 2012-05-10 01:40 8953 s3://paramesh-test/block_-6515202635859543492 2012-05-10 02:20 48051 s3://paramesh-test/block_1132982570595970987 2012-05-10 01:40 48052 s3://paramesh-test/block_3632190765594848890 2012-05-10 02:20 1160 s3://paramesh-test/block_363439138801598558 2012-05-10 01:40 1160 s3://paramesh-test/block_3786390805575657892 2012-05-10 01:40 11876 s3://paramesh-test/block_4393980661686993969 ```