Spring JDBC support and large dataset
java, jdbc, oracle, spring
Solution
The Oracle JDBC driver has proper support for the `setFetchSize()` method on `java.sql.Statement`, which allows you to control how many rows the driver will fetch in one go.
However, `RowMapper` as used by Spring works by reading each row into memory, getting the `RowMapper` to translate it into an object, and storing each row's object in one big list. If your result set is huge, then this list will get big, regardless of how JDBC fetches the row data.
If you need to handle large result sets, then RowMapper isn't scaleable. You might consider using `RowCallbackHandler` instead, along with the corresponding methods on JdbcTemplate. `RowCallbackHandler` doesn't dictate how the results are stored, leaving it up to you to store them.
Problem
When using one of the various JDBC template methods I am confused on how to iterate/scroll over large result sets (which won't fit into memory). Even without a direct exposure of an Iterable interface I would at least expect instances of RowCallbackHandler to get called while the query is executing not after it's finished (or the heap overfloats). I did have a look a at this (which changed nothing for me despite being similar in spirit to this post on stack overflow) and at this post in the spring forums. The latter seems to suggest that the callback handler should indeed get called while the cursor is fetching data. My tests however show no such behaviour. The database is an Oracle10g. I am using the 11.1.0.7.0-Production driver and Spring 2.5.6.SEC01. Any ideas anyone how to iterate over result sets, preferably while keeping the mapping logic of RowMapper etc.?