Threading and Queues in Python3: sentinel value in input_queue fails to break out of while loop

General Overview: I am attempting to build a trivial example of an application that uses threads to scrape information from the web. I know there are dedicated modules for this (scrapy, for instance), but I wanted to do it more to learn and understand how threading works and to understand the pitfalls. In addition, I've seen various tutorials (IBM tutorial and others), questions on SO, and even some recipes in the Python Cookbook, 3rd Ed. that describe how to do it and yet I'm still getting hung up somewhere when using threading/queues. Firstly, I have read on stackoverflow (and in the Cookbook) that it's a waste to subclass `threading.Thread`, so I've been trying to do it using `Thread(target=..., args=(...))`. Most tutorials appear to use the (older?) subclassing method, and I may be getting confused by going in this newer(?) way. Second, in the Python Cookbook, they offer the example of placing a `_sentinel` value into a work queue, which your methods watch out for and break out of if they discover it. I think this is where I'm having trouble. Finally, I am sure I must be engaging in all kinds of overkill, so I would appreciate tips or pointers on where I am overcomplicating things. Thanks for your suggestions. Specific Coding Problem and Attempts: I have a work_queue of (company name, url) tuples. My method is parsing the urls for the stock ticker name and sending a tuple (ticker name, company name) to an output queue. I am trying to use a sentinel to break out of the method when all values have been consumed: ``` def grab_ticker_name(input_q, output_q, _sentinel): '''Function to be threaded later: grabs company ticker from url and sends it to an output queue.''' while True: item = input_q.get() if item is _sentinel: print("We encountered the sentinel!!") input_q.put(_sentinel) # See page 492 of Python Cookbook, 3rd Ed. break else: company_name, company_url = item with urlopen(company_url) as f: newsoup = BeautifulSoup(f) if len(newsoup.select('.inlink_chart > a')) > 0: ticker_name = newsoup.select('.inlink_chart > a')[0].string output_q.put((ticker_name, company_name)) print("grab_ticker_name grabbed the following: {ticker_name}, {company_name}".format_map(vars())) input_q.task_done() ``` I am populating the queue and building threads like so (with `print` statements just to help me see what's going on): ``` def main(): work_queue = queue.Queue() out_queue = queue.Queue() _sentinel = object() # This recommendation comes from the *Python Cookbook* stock_list = StockBuilder() print("Populating the work_queue now") for item in stock_list.yield_url(): work_queue.put(item) work_queue.put(_sentinel) t1 = threading.Thread(target=grab_ticker_name, args=(work_queue, out_queue, _sentinel)) t2 = threading.Thread(target=grab_ticker_name, args=(work_queue, out_queue, _sentinel)) print("Now we're going to start our threads!") t1.start() ; t2.start() print("Now we're going to attempt to join our work queue.") work_queue.join() ``` It appears to work and gives me row upon row of values, but then appears to loop infinitely until I have to cancel it (CTRL+C): ``` ... grab_ticker_name grabbed the following: MRK, Merk grab_ticker_name grabbed the following: HUM, Humana We encountered the sentinel!! grab_ticker_name grabbed the following: WLP, WellPoint We encountered the sentinel!! ^CTraceback (most recent call last). ... ``` Question: Why doesn't the `break` expression make it so the function returns to main()? Finally, I also would like to be able to take all data from the `out_queue` and put it in another data structure (list, dictionary, whatever). Question 2: I know that I may use a list with locks for this, but I am also confused here because I thought I had read that a queue was an easier way for threads to modify a mutable object? Question 2a: Assuming I use the out_queue, is there an easy way once the `out_queue` has been populated to simply grab all values from it? I thought maybe I would put another `_sentinel` into the `out_queue` after `work_queue.join()` and then do another `while` loop and work on each item in `out_queue` if it's not `_sentinel`? Is this a naive way to do this? Thanks for reading.