AWS S3 Client for linux with multipart upload

Question

What amazon s3 client do you use in linux with multipart upload feature? I have 6GB of zip files to upload and s3curl is not possible due to maximum limit of 5GB only. Thanks. James

Accepted Answer

I use S3 Tools, it will automatically use the multipart upload feature for files larger than 15MB for all PUT commands:

Multipart is enabled by default and kicks in for files bigger than 15MB. You can set this treshold as low as 5MB (Amazon’s limit) with —multipart-chunk-size-mb=5 or to any other value between 5 and 5120 MB

Once installed and configured, just issue the following command:

```
~$ s3cmd put largefile.zip s3://bucketname/largefile.zip
```

Alternatively, you could just use split from the command-line on your zip file:

```
split -b1024m largefile.zip largefile.zip-
```

and recombine later on your filesystem using:

```
cat largefile.zip-* > largefile.zip
```

If you choose the second option, you may want to store MD5 hashes of the files prior to upload so you can verify the integrity of the archive when it's recombined later.

AWS S3 Client for linux with multipart upload

Solution

Problem