Atomic write to multiple files
atomic, database, filesystems, python, synchronization
Solution
Here is what I'm thinking. First ensure open is synchronized, then perform the following:
- Write to temp files: file1~, file2~ and special file success~ (must be written first).
- After a successful write, remove file success~.
- Rename the files to file1 and file2.
If something breaks:
- Check if success~ exists.
- If it does, do not bother repairing. A rollback was implicitly performed because the files were not updated (no renames).
- If success~ does not exist, things broke after writing and during renaming. In this case repairing is as simple as renaming filex~ to filex.
Problem
Suppose I have a set of files. How do I ensure that writing to all these files is atomic. I thought about writing to temp files and only after the writing is successful, perform an atomic rename of each file. However renaming all the files at once isn't atomic. Also this will not scale to very large files if we'd like to append to these files. Instead I thought about implementing transactions but then that's becoming a project on its own. I realize that this is pretty much about implementing a mini database. How would you do it in Python? ``` d = FileWriter.open(['file1', 'file2'], 'wb+') d.write('add hello world to files') d.close() ``` Ensure that d.write is atomic or at least rollback to original files if unsuccessful.