Is it safe to use MD5 checksums to search for duplicate files across multiple hard drives?
checksum, md5, python
Solution
The probability of finding an md5 collision between two files by accident is:
0.000000000000000000000000000000000000002938735877055718769921841343055614194546663891
the probability of getting hit by 15km size asteroid is `0.00000002`. I'd say yes.
Backing up the files and well testing the script remains a good advice, human mistakes and bugs are more luckily to happen.
Problem
I've been tasked with consolidating about 15 years of records from a laboratory, most of which is either student work or raw data. We're talking 100,000+ human-generated files. My plan is to write a Python 2.7 script that will map the entire directory structure, create checksums for each, and then flag duplicates for deletion. I'm expecting probably 10-25% duplicates. My understanding is that MD5 collisions are possible, theoretically, but so unlikely that this is essentially a safe procedure (let's say that if 1 collision happened, my job would be safe). Is this a safe assumption? In case implementation matters, the only Python libraries I intend to use are: - `hashlib` for the checksum; - `sqlite` for databasing the results; - `os` for directory mapping