Hibernate queries much slower with flushMode=AUTO until clear() is called

Question

I have a long running (but rather simple) application that uses Hibernate (via JPA). It was experiencing rather dramatic slowdown as it ran. I've been able to narrow down to requiring an occasional `entityManager.clear()` call. When Hibernate's entity manager is tracking 100,000 entities, it is ~100x slower than when it is tracking only a few (see results below). My question is: why does Hiberate slow down so much when it is tracking a lot of entities? And are there any other ways around it? !!! Update: I've been able to narrow this down to Hibernate's auto-flushing code. !!! Specifically to `org.hibernate.event.internal.AbstractFlushingEventListener`'s `flushEntities()` method (at least in Hibernate 4.1.1.Final). In it there is a loop that iterates over ALL entities in the persistence context, performing some extensive checks around flushing each of them (even though all the entities are already flushed in my example!). So partially answering the second part of my question, the perfo…

Accepted Answer

But mainly I'm curious why Hibernate seems to exhibit O(n) or even O(n^2) lookups for the queries--seems like it should be able to use a hashtable or binary-tree under the hood to keep the queries fast. Notice the 2-orders-of-magnitude difference when its tracking 100000 entities vs 100 entities.

The O(n²) complexity results from the way a query must be handled. Since Hibernate internally deferred updates and inserts as long as it can (to use the chance to group similar updates/insertions together, especially if you set multiple properties of an object).

So before you can safely query objects in the database, Hibernate has to detect all object changes and flush all changes. The problem here is that hibernate has also some notification and interception going on. So It iterates over every entity object managed by the persistence context. Even if the object is in itself is not mutable it might hold mutable objects or even reference collections.

Also the interception mechanism grants you access to any object being considered for being dirty to allow your own code to implement additional dirtyness checks or perform additional calculations like calculate sums, average values, log additional information and so on.

But let's look at the code for a minute:

The flush call to prepare the query results in:

```
DefaultFlushEventListener.onFlush(..)
  -> AbstractFlushingEventListener.flushEverythingToExecution(event)
    -> AbstractFlushingEventListener.prepareEntityFlushes(..)
```

The implementation uses:

```
for ( Map.Entry me : IdentityMap.concurrentEntries( persistenceContext.getEntityEntries() ) ) {
        EntityEntry entry = (EntityEntry) me.getValue();
        Status status = entry.getStatus();
        if ( status == Status.MANAGED || status == Status.SAVING || status == Status.READ_ONLY ) {
            cascadeOnFlush( session, entry.getPersister(), me.getKey(), anything );
        }
    }
```

As you can see a map of all entities in the persistence context is retrieved and iterated.

That means for every invocation of a query you iterate over all former results to check for dirty objects. And even more cascadeOnFlush creates a new Object and does even more things. Here is the code of cascadeOnFlush:

```
private void cascadeOnFlush(EventSource session, EntityPersister persister, Object object, Object anything)
throws HibernateException {
    session.getPersistenceContext().incrementCascadeLevel();
    try {
        new Cascade( getCascadingAction(), Cascade.BEFORE_FLUSH, session )
        .cascade( persister, object, anything );
    }
    finally {
        session.getPersistenceContext().decrementCascadeLevel();
    }
}
```

So this is the explanation. Hibernate just checks every object managed by the persistence context every time you issue a query.

So for everyone reading this here is the complexity calculation:

- Query: 0 entities

- Query: 1 entity

- Query: 2 entities ..

- Query: 100 entities . .. 100k + 1 Query: 100k entries

So we have O(0+1+2...+n) = O(n(n+1)/2) = O(n²).

This explains your observation. In order to maintain a small cpu and memory footprint hibernate managed persistence context should be kept as small as possible. Letting Hibernate manage more than lets say 100 or 1000 entities slows down Hibernate considerably. Here one should consider to change the flush mode, use a second session for query and one for change (if this is possible at all) or use StatelessSession.

So your observation is correct it is O(n²) going on.

Hibernate queries much slower with flushMode=AUTO until clear() is called

Solution

Problem

Related problems