python, how to incrementally create Threads
multithreading, python
Solution
First of all you join only the last thread. There is no guarantee that it will be finished the last. You should use like that:
from time import sleep
delay = 0.5
tlist = [threading.Thread(target=search_sl, args=(ra,dec, q)) for ra, dec in candidates ]
map(lambda t:t.start(), tlist)
while(any(map(lambda t:t.isAlive()))): sleep(delay)
The second issue is the running 60K threads at the moment requires really huge hardware resource :-) It's better to queue your tasks and then process by workers. The number of worker threads must be limited. Like that (haven't tested the code, but the idea is clear I hope):
from Queue import Queue
from threading import Thread
from time import sleep
tasks = Queue()
map(tasks.put, candidates)
maxthreads = 50
delay = 0.1
try:
threads = [Thread(target=search_sl, args=tasks.get()) \
for i in xrange(0,maxthreads) ]
except Queue.Empty:
pass
map(lambda t:t.start(), threads)
while not tasks.empty():
threads = filter(lambda t:t.isAlive(), threads)
while len(threads) < maxthreads:
try:
t = Thread(target=search_sl, args=tasks.get())
t.start()
threads.append(t)
except Queue.Empty:
break
sleep(delay)
while(any(map(lambda t:t.isAlive(), threads))): sleep(delay)
Problem
I have a list of items aprox 60,000 items - i would like to send queries to the database to check if they exist and if they do return some computed results. I run an ordinary query, while iterating through the list one-by-one, the query has been running for the last 4 days. I thought i could use the threading module to improve on this. I did something like this ``` if __name__ == '__main__': for ra, dec in candidates: t = threading.Thread(target=search_sl, args=(ra,dec, q)) t.start() t.join() ``` I tested with only 10 items and it worked fine - when i submitted the whole list of 60k items, i run into errors i.e, "maximum number of sessions exceeded". What I want to do is to create maybe 10 thread at a time. When the 1st bunch of thread have finished excuting, i send another request and so on.