Threading in python using queue
multithreading, python
Solution
Setting the thread's to be `daemon` threads causes them to exit when the main is done. But, yes you are correct in that your threads will run continuously for as long as there is something in the `queue` else it will block.
The documentation explains this detail Queue docs
The python Threading documentation explains the `daemon` part as well.
The entire Python program exits when no alive non-daemon threads are left.
So, when the queue is emptied and the `queue.join` resumes when the interpreter exits the threads will then die.
EDIT: Correction on default behavior for `Queue`
Problem
I wanted to use threading in python to download lot of webpages and went through the following code which uses queues in one of the website. it puts a infinite while loop. Does each of thread run continuously with out ending till all of them are complete? Am I missing something. ``` #!/usr/bin/env python import Queue import threading import urllib2 import time hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com", "http://ibm.com", "http://apple.com"] queue = Queue.Queue() class ThreadUrl(threading.Thread): """Threaded Url Grab""" def __init__(self, queue): threading.Thread.__init__(self) self.queue = queue def run(self): while True: #grabs host from queue host = self.queue.get() #grabs urls of hosts and prints first 1024 bytes of page url = urllib2.urlopen(host) print url.read(1024) #signals to queue job is done self.queue.task_done() start = time.time() def main(): #spawn a pool of threads, and pass them queue instance for i in range(5): t = ThreadUrl(queue) t.setDaemon(True) t.start() #populate queue with data for host in hosts: queue.put(host) #wait on the queue until everything has been processed queue.join() main() print "Elapsed Time: %s" % (time.time() - start) ```