Inserting 1000000 rows in sqlite3 database

android, sqlite

Solution

Quick example time why you should do the right thing instead of "wrong". This was tested running on ICS 4.0.4, which has horrible INSERT-performance.

First, a simple `SQLiteOpenHelper` that creates a table with a `UNIQUE` constraint on a column to cause conflicts now-and-then.

class SimpleHelper extends SQLiteOpenHelper {
    // InsertHelpers are a really good idea - they format a prepared statement
    // for you automatically.
    InsertHelper mInsert;
    public SimpleHelper(Context context) {
        super(context, "tanika.db", null, 1);
    }
    @Override
    public void onOpen(SQLiteDatabase db) {
        super.onOpen(db);
        mInsert = new InsertHelper(db, "target");
    }
    @Override
    public void onCreate(SQLiteDatabase db) {
        db.execSQL("CREATE TABLE target (\n" +
                "_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,\n" +
                "val1 TEXT NOT NULL,\n" +
                "val2 TEXT NOT NULL,\n" +
                "val3 TEXT NOT NULL,\n" +
                // Let's make one unique so we can get some juicy conflicts
                "val4 TEXT NOT NULL UNIQUE\n" +
                ")");
    }
    @Override
    public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
    }
}

Bundled in any old `Activity` we add the following simple test method:

long test(final int n) {
    long started = System.currentTimeMillis();
    ContentValues values = new ContentValues();

    for (int i = 0; i < n; i++) {
        values.clear();
        // Every 20th insert, generate a conflict in val4
        String val4 = String.valueOf(started + i);
        if (i % 20 == 0) {
            val4 = "conflict";
        }
        values.put("val1", "Value1");
        values.put("val2", "Value2");
        values.put("val3", "Value3");
        values.put("val4", val4);
        mHelper.mInsert.replace(values);
    }
    return System.currentTimeMillis() - started;
}

As you can see, this would cause a conflict every 20th `INSERT` or so. Calling `InsertHelper#replace(..)` causes the helper to use a `INSERT OR REPLACE` on conflicts.

Now, let's run this test code with & without a transaction surrounding it.

class Test1 extends AsyncTask<Integer, Void, Long> {
    @Override
    protected Long doInBackground(Integer... params) {
        return test(params[0]);
    }
    @Override
    protected void onPostExecute(Long result) {
        System.out.println(getClass().getSimpleName() + " finished in " + result + "ms");
    }
}

class Test2 extends AsyncTask<Integer, Void, Long> {
    protected Long doInBackground(Integer... params) {
        SQLiteDatabase db = mHelper.getWritableDatabase();
        db.beginTransaction();
        long started = System.currentTimeMillis();
        try {
            test(params[0]);
            db.setTransactionSuccessful();
        } finally {
            db.endTransaction();
        }
        return System.currentTimeMillis() - started;
    }
    @Override
    protected void onPostExecute(Long result) {
        System.out.println(getClass().getSimpleName() + " finished in " + result + "ms");
    }
}

Everything is started like this:

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.main);

    mHelper = new SimpleHelper(this);
    mHelper.getWritableDatabase(); // Forces the helper to initialize.
    new Test1().execute(2055);
    new Test2().execute(2055);
}

And the results? Without a transaction the `INSERT`s take 41072ms. With transactions they take 940ms. In short, FFS, start using `InsertHelper`s and transactions.

Problem

Original source