Database disk image is malformed from many concurrent writes
python, sqlite
Solution
SQLite does not support storage on a distributed file system. Concurrent access required locking and that is not transferred across such a system.
You'll have to move to a database that supports a networked model instead, such as MySQL or PostgreSQL.
Problem
I'm running many (18k) batch jobs on a cluster with a Lustre filesystem. The jobs are submitted at the same time, each takes about 3 seconds, and they write the result using the `sqlite3` python module. The write part of the code is very simple: ``` with sqlite3.connect(name, timeout=900) as conn: conn.execute( "insert into someTable values (?, ?)", (value1, value2)) ``` but a lot of the jobs will throw an exception: ``` sqlite3.DatabaseError: database disk image is malformed ``` and sometimes ``` sqlite3.OperationalError: unable to open database file ``` I'm guessing this has something to do with lots of jobs putting a lock on the file when they write to it, but my impression was that `sqlite3` should know to wait patiently for the file to be free. Is my error likely a result of too many concurrent writes? How can I fix it?