gsutil cannot copy to s3 due to authentication
amazon-s3, google-cloud-storage, gsutil
Solution
As per https://issuetracker.google.com/issues/62161892, `gsutil` v4.28 does support `AWS` v4 signatures by adding to `~/.boto` a new `[s3]` section like
[s3]
# Note that we specify region as part of the host, as mentioned in the AWS docs:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
host = s3.eu-east-2.amazonaws.com
use-sigv4 = True
The use of that section is inherited from `boto3` but is currently not created by `gsutil config` so it needs to be added explicitly for the target endpoint.
For s3-to-GCS, I will consider the more server-less `Storage Transfer Service` API.
Problem
I need to copy many (1000+) files to s3 from GCS to leverage an AWS lambda function. I have edited `~/.boto.cfg` and commented out the 2 aws authentication parameters but a simple `gsutil ls s3://mybucket` fails from either an GCE or EC2 VM. Error is `The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.`. I use gsutil version: `4.28` and locations of GCS and S3 bucket are respectively `US-CENTRAL1` and `US East (Ohio)` - in case this is relevant. I am clueless as the AWS key is valid and I enabled http/https. Downloading from GCS and uploading to S3 using my laptop's Cyberduck is impracticable (>230Gb)