python: Mac OS X. malloc error. pointer being freed was not allocated. abort trap 6

Question

I am running a multithreading python script. what it does is it will crawl the web and insert/update into mysql. here is my code mythread.py ``` import threading import time class MyThread (threading.Thread): def __init__(self, threadID, threadname, q): threading.Thread.__init__(self) self.threadID = threadID self.threadname = threadname self.queue = q self.__exitFlag = False self.__signal_lock = threading.Lock() def run(self): print "Starting " + self.threadname self.process_data() print "Exiting " + self.threadname def stop(self): with self.__signal_lock: self.__exitFlag = True def process_data(self): while not self.__exitFlag: if not self.queue.empty(): data = self.queue.get() # crawl data from the web... # update to mysql # assuming we have already connected mysql: # db = MySQLDb() # db.connect query = "" db.query(query) ``` mysql_db.py ``` class MySQLDb: conn = None def connect(self): self.conn = MySQLdb.connect( host="127.0.0.1", user = "root", passwd = "password", db = "moviest…

Accepted Answer

I ran in to the same malloc error on OSX using MySQLdb. The thing that was causing my error was that was sharing MySQLdb connections between threads. Using a connection per thread fixed it for me.

From the documentation http://mysql-python.sourceforge.net/MySQLdb.html:

The MySQL protocol can not handle multiple threads using the same connection at once. Some earlier versions of MySQLdb utilized locking to achieve a threadsafety of 2. While this is not terribly hard to accomplish using the standard Cursor class (which uses mysql_store_result()), it is complicated by SSCursor (which uses mysql_use_result(); with the latter you must ensure all the rows have been read before another query can be executed. It is further complicated by the addition of transactions, since transactions start when a cursor execute a query, but end when COMMIT or ROLLBACK is executed by the Connection object. Two threads simply cannot share a connection while a transaction is in progress, in addition to not being able to share it during query execution. This excessively complicated the code to the point where it just isn't worth it.

The general upshot of this is: Don't share connections between threads. It's really not worth your effort or mine, and in the end, will probably hurt performance, since the MySQL server runs a separate thread for each connection. You can certainly do things like cache connections in a pool, and give those connections to one thread at a time. If you let two threads use a connection simultaneously, the MySQL client library will probably upchuck and die. You have been warned.

python: Mac OS X. malloc error. pointer being freed was not allocated. abort trap 6

Solution

Problem