uploading and compressing file to s3

amazon-s3, amazon-web-services, compression, gzip, upload

Solution

In the linux shell, via aws-cli, this was added about 3 months after you asked the question :-)

Added the ability to stream data using `cp`

So the best you can do, I guess, is to pipe the output of gzip to aws cli:

Upload from stdin:

`gzip -c big_file | aws s3 cp - s3://bucket/folder/big_file.gz`

Download to stdout:

`aws s3 cp s3://bucket/folder/big_file.gz - | gunzip -c ...`

Problem

I've recently started working with S3 and have come across this need to upload and compress large files (10 GB +-) to S3. The current implementation I'm working with is creating a temporary compressed file locally and then uploading it to S3 and finally deleting the temporary file. For a 10 GB file, I have almost 20 GB locally stored until the upload is done. I need a way to transfer the file to s3 and then compress it there. Is this approach viable? If yes, how should I address it? If not, is there any way I can minimize the local space needed? I've seen someone suggesting that the file could be uploaded to the S3, downloaded to an EC2 in the same region, compressed there and then uploaded back to the S3 while deleting the first copy on S3. This might work but it seems that 2 uploads for getting one file up wouldn`t be an advantage cost-wise. I've tried to upload a compression stream without success but I`ve just discovered S3 does not support compression streaming and now I am clueless as to how to proceed. I'm using the gzip library on .NET

Original source