Storing a large number of images
filesystems, image, tree
Solution
We had a similar problem in the past. And found a nice solution:
- Give each image an unique guid.
- Create a database record for each image containing the name, location, guid and possible location of sub images (thumbnails, reducedsize, etc.).
- Use the first (one or two) characters of the guid to determine the toplevel folder.
- If the folders have too much files, split again. Update the references and you are ready to go.
- If the number of files and the accesses are too high, you can spread folders over different file servers.
We have experienced that using the guids, you get a more or less uniform division. And it worked like a charm.
Links which might help to generate a unique ID:
- http://en.wikipedia.org/wiki/Universally_Unique_Identifier
- http://en.wikipedia.org/wiki/Sha1
Problem
I'm thinking about developing my own PHP based gallery for storing lots of pictures, maybe in the tens of thousands. At the database I'll point to the url of the image, but here's the problem: I know is impractical to have all of them sitting at the same directory in the server as it would slow access to a crawl, so, how would you store all of them? Some kind of tree based on the name of the jpeg/png? What rules to partition the images would you recommend me? (It will focused for using in cheapo dot coms, so no mangling with the server is possible)