Best way to store large amount of data of users

Question

I store files of users in their own name directory something like ``` /username/file01.jpg /username/file02.mp4 /username/file03.mp3 ``` But if more users come and upload more files then this creates problem because this will lead to migration of some or many users to another drive.I choose username directory solution first because i dont want filenames to be mixed. I dont want to change filename too. Also if another user upload same filename then it creates problem ,if the files are stored with original name. What could be the best way to do this. I have one solution but want to ask community is this the best way . i will use sequential folders and then hash the file name to some thing very unique and store into the directory. What i will do is store the original name of file and username into database and hashvalue of filename which is stored in Disk. When anyone want to access that file,I will read that file through php either replace the name or will do something at that point so…

Accepted Answer

I handle file metadata on the database and retrive the files with a UUID. What i do is:

- Content based identification

- MD5 from file's content

- Namespaced UUID:v5 to generate unique identifier based on user's uuid and file's md5.

- Custom function to generate path based on 'realname'.

- Save on the database: uuid, originalname (the uploaded name), realname (the generated name), filesize, and mime. (optional dateAdded, and md5)

- File retrival.

- UUID to retrive metadata.

- regenerate filepath based on realname.

- Originalname is used to show a familiar name to the user that downloads the file.

I process the file's name assigning it a namespaced UUID as the database primary key, and Generate the path based on User and filename. The precondition is that your user has a uuid assigned to him. The following code will help you avoid id collisions on the database, and help you identify files by its contents (If you ever need to have a way to spot duplicate content and not necesarily filenames).

```
$fileInfo = pathinfo($_FILE['file']['name']);
$extension = (isset($fileInfo['extension']))?".".$fileInfo['extension']:"";

$md5Name = md5_file($_FILE['file']['tmp_name']); //you could use other hash algorithms if you are so inclined.

$realName = UUID::v5($user->uuid, $md5Name) . $extension; //UUID::v5(namespace, value).
```

I use a function to generate the filepath based on some custom parameteres, you could use $username and $realname. This is helpful if you implement a distributed folder structure which you might have partitioned on file naming scheme, or any custom scheme.

```
function generateBasePath($realname, $customArgsArray){
    //Process Args as your requirements.
    //might as well be  "$FirstThreeCharsFromRealname/"
    //or a checksum that helps you decide which drive/volume/mountpoint to use.
    //like some files on the local disk and some other from an Amazon::S3 mountpoint.
    return $mountpoint.'/'.$generatedPath; 
}
```

As an added bonus this also:

- helps you maintain a versioned file repository if you add an attribute on the file's record of which file (uuid) it has replaced.

- create a application Access Control List if you add an attributes of 'owner' and/or 'group'

- also works on a single folder structure.

Note: I used php's $_FILE as an example of the file source based on this question's tags. It can be from any file source or generated content.

Best way to store large amount of data of users

Solution

Problem