Python Interpreter blocks Multithreaded DNS requests?
multithreading, network-programming, python
Solution
On some systems, getaddrinfo is not thread-safe. Python believes that some such systems are FreeBSD, OpenBSD, NetBSD, OSX, and VMS. On those systems, Python maintains a lock specifically for the netdb (i.e. getaddrinfo and friends).
So if you can't switch operating systems, you'll have to use a different (thread-safe) resolver library, such as twisted's.
Problem
I just played around a little bit with python and threads, and realized even in a multithreaded script, DNS requests are blocking. Consider the following script: from threading import Thread import socket ``` class Connection(Thread): def __init__(self, name, url): Thread.__init__(self) self._url = url self._name = name def run(self): print "Connecting...", self._name try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setblocking(0) s.connect((self._url, 80)) except socket.gaierror: pass #not interested in it print "finished", self._name if __name__ == '__main__': conns = [] # all invalid addresses to see how they fail / check times conns.append(Connection("conn1", "www.2eg11erdhrtj.com")) conns.append(Connection("conn2", "www.e2ger2dh2rtj.com")) conns.append(Connection("conn3", "www.eg2de3rh1rtj.com")) conns.append(Connection("conn4", "www.ege2rh4rd1tj.com")) conns.append(Connection("conn5", "www.ege52drhrtj1.com")) for conn in conns: conn.start() ``` I dont know exactly how long the timeout is, but when running this the following happens: - All Threads start and I get my printouts - Every xx seconds, one thread displays finished, instead of all at once - The Threads finish sequentially, not all at once (timeout = same for all!) So my only guess is that this has to do with the GIL? Obviously the threads do not perform their task concurrently, only one connection is attempted at a time. Does anyone know a way around this? (asyncore doesnt help, and I'd prefer not to use twisted for now) Isn't it possible to get this simple little thing done with python? Greetings, Tom edit: I am on MacOSX, I just let my friend run this on linux, and he actually does get the results I wished to get. His socket.connects()'s return immediately, even in a non Threaded environment. And even when he sets the sockets to blocking, and timeout to 10 seconds, all his Threads finish at the same time. Can anyone explain this?