Amazon AWS S3 file naming strategy for performance
amazon-s3, amazon-web-services, file, naming, naming-conventions
Solution
If you won't use some unique string when building a key, you'll reach very soon the problem of a key override.
dsca007.jpg is not very unique. there is a big chance that a file with the same name will be uploaded to your s3 bucket. In this case you'll lost the old file or get an error, depends on your configuration.
e.g. this is what we are using: `https://s3.amazonaws.com/bucket_name/user_media/videos/screenshots/cmXRyLRQxe9R139023426817_vid001.jpeg`
where the prefix: cmXRyLRQxe9R139023426817_ is a self generated string we build and concat to the original file name: vid001.jpg before uploading to s3.
Problem
I am currently utilizing the following naming scheme: ``` /#{bucket_name}/#{customer_name}/fi/le/na/filename.jpg ``` So an image for a file named `dsca007.jpg` for customer `bent` would be stored here: ``` /images/bent/ds/ca/00/dsca007.jpg ``` But I can understand that S3 wants a different form like this: Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html Source: https://www.youtube.com/watch?v=uXHw0Xae2ww#t=535 I understand that they want me to change my naming scheme to something like: ``` /#{bucket_name}/fi/le/na/#{customer_name})filename.jpg ``` But then he shows this slide in the youtube video: Does that mean that my first approach was fine? I really liked the first structure better because I was able to get a list of files owned by one customer, is there a way to list all files under a customer with the last mentioned naming scheme? It was also quite easy to figure out how much storage each customer was using by running this command: ``` s3cmd du s3://images/#{customer_name} ``` How can I do that with the new naming scheme?