Get all versions of an object in an AWS S3 bucket?

amazon-s3, amazon-web-services, key, version

Solution

So, I ran into this brick wall this morning. This seemingly trivial thing is incredibly difficult to do, it turns out.

The API you want is the GET Bucket Object versions API, but it is sadly non-trivial to use.

First, you have to steer clear of some non-solutions: `KeyMarker`, which is documented by `boto3` as,

KeyMarker (string) -- Specifies the key to start with when listing objects in a bucket.

…does not start with the specified key when listing objects in a bucket; rather, it starts immediately after that key, which makes it somewhat useless here.

The best restriction this API provides is `Prefix`; this isn't going to be perfect, since there could be keys that are not our key of interest that nonetheless contain our key.

Also beware of `MaxKeys`; it is tempting to think that, lexicographically, our key should be first, and all keys which have our key as a prefix of their key name would follow, so we could trim them using `MaxKeys`; sadly, `MaxKeys` controls not how many keys are returned in the response, but rather the number of versions. (And I'm going to presume that isn't known in advance.)

So, `Prefix` is the best it seems that can be done. Also note that, at least in some languages, the client library will not handle pagination for you, so you'll additionally need to deal with that.

As an example in `boto3`:

response = client.list_object_versions(
    Bucket=bucket_name, Prefix=key_name,
)
while True:
    # Process `response`
    ...
    # Check if the results got paginated:
    if response['IsTruncated']:
        response = client.list_object_versions(
            Bucket=bucket_name, Prefix=key_name,
            KeyMarker=response['NextKeyMarker'],
            VersionIdMarker=response['NextVersionIdMarker'],
        )
    else:
       break

Problem

I've enabled object versioning on a bucket. I want to get all versions of a key inside that bucket. But I cannot find a method go do this; how would one accomplish this using the S3 APIs?

Original source