SQLAlchemy memory leak when instrumented objects not commited (e.g. rollbacked)
sqlalchemy
Solution
Using `__del__()` can create memory leaks, because if an object becomes the subject of a cycle, it is then unreachable by cyclic GC. That is the case in this test because the SQLAlchemy object instrumentation creates a cycle from the object to itself in the case that the object is "dirty", that is, has pending attributes to be flushed to the database, so that you can add a dirty object to the Session and then lose all references to it, and the changes will still be flushed. This "reference marker" is removed as soon as the object is marked as clean (i.e. flushed).
For SQLAlchemy 0.8.1 I've improved this behavior: one is that this reference cycle is no longer created for "pending" or "detached" objects, that is, objects which aren't associated with a Session. Instead, the object is checked when it is attached to a Session for the .modified flag, and the reference marker is associated only at that point (and is removed when the object becomes clean, as was the case already). If the object is detached from the session, the marker is unconditionally removed, even if the object still has changes - the .modified flag stays true.
Additionally, I've added a warning when a class is first mapped and detected as having a `__del__()` method. It's very easy for a Python object to have cycles, and in the case of SQLAlchemy the pattern of placing a `relationship()` on a class with a backref will also have the effect of creating reference cycles, so even with the state management improvement here, using `__del__()` is a bad idea.
Problem
resolved not an issue, was caused by the presence of: ``` def __del__(self): print "deleted!" ``` in the model class. As soon as I removed it from the model class, I cannot experiment any problems with memory usage. I have some models that I am using with an ad-hoc session/engine and this is working fine, but when I want to create these objects outside of a database/session context, it seems SQLAlchemy instrumentation keeps a reference on the objects and they are never deleted. What I would like to do is to create a model object like a "normal" Python object and never add it to any session/database (but in other contexts I need to add it to a database). Is this wrong? ``` import gc from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, String base = declarative_base() class SomeObject(base): __tablename__ = "SomeObject" instrumented_name = Column('name', String(55), primary_key=True) uninstrumented_name = None def __del__(self): print "deleted!" obj1 = SomeObject() obj1.uninstrumented_name = "foo" obj1 = None #obj1 is properly deleted obj2 = SomeObject() obj2.instrumented_name = "bar" obj2 = None gc.collect() #obj2 never deleted ``` Edit I did some additional testing and it seems SQLAlchemy will cause memory leak if the objects are never commited into a session (e.g. rollbacked) Is there a method that will force SQLAlchemy to release it's references on instrumented objects? ``` import gc from sqlalchemy.orm import scoped_session, sessionmaker from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, Integer, String, create_engine Base = declarative_base() class MemoryMonster(Base): __tablename__ = "MemoryMonster" _id = Column('id', Integer(), primary_key=True) _name = Column('name', String(55)) def __init__(self): self._name = "some monster name" self._eat_some_ram = ' ' * 1048576 def __del__(self): print "deleted!" engine = create_engine("sqlite:///:memory:") session_factory = sessionmaker(engine) Session = scoped_session(session_factory) Base.metadata.create_all(engine) def create_and_commit(): session = Session() for _ in range(100): session.add(MemoryMonster()) session.commit() Session.remove() gc.collect() def create_and_rollback(): session = Session() for _ in range(100): monster = MemoryMonster() session.add(monster) session.expunge(monster) session.rollback() Session.remove() gc.collect() def create_do_not_include_in_session(): session = Session() for _ in range(100): monster = MemoryMonster() session.rollback() Session.remove() gc.collect() # Scenario 1 - Objects included in the session and commited # No memory leak create_and_commit() # Scenario 2 - Objects included in the session and rollbacked # Memory leak create_and_rollback() # Scenario 3 - Objects are not not included in the session # Memory leak create_do_not_include_in_session() ```