efficient python raw_input and serial port polling

multithreading, polling, python, raw-input, serial-port

Solution

You're seeing the high CPU usage because your main thread is using the non-blocking `get_nowait` call to poll two different queues in an infinite loop, which means most of the time your loop is going to be constantly looping. Constantly running through the loop uses CPU cycles, just as any tight infinite loop does. To avoid using lots of CPU, you want to have your infinite loops use blocking I/O, so that they wait until there's actually data to process before continuing. This way, you're not constantly running through the loop, and therefore using CPU.

So, user input thread:

while True:
    data = raw_input() # This blocks, and won't use CPU while doing so
    queue.put({'type' : 'input' : 'data' : data})

COM thread:

while True:
    data = com.get_com_data()  # This blocks, and won't use CPU while doing so
    queue.put({'type' : 'COM' : 'data' : data})

main thread:

while True:
    data = queue.get() # This call will block, and won't use CPU while doing so
    # process data

The blocking `get` call will just wait until it's woken up by a `put` in another thread, using a `threading.Condition` object. It's not repeatedly polling. From `Queue.py`:

# Notify not_empty whenever an item is added to the queue; a
# thread waiting to get is notified then.
self.not_empty = _threading.Condition(self.mutex)

...

def get(self, block=True, timeout=None):
    self.not_empty.acquire()
    try:
        if not block:
            if not self._qsize():
                raise Empty
        elif timeout is None:
            while not self._qsize():
                self.not_empty.wait()  # This is where the code blocks
        elif timeout < 0:
            raise ValueError("'timeout' must be a non-negative number")
        else:
            endtime = _time() + timeout
            while not self._qsize():
                remaining = endtime - _time()
                if remaining <= 0.0:
                    raise Empty
                self.not_empty.wait(remaining)
        item = self._get()
        self.not_full.notify()
        return item
    finally:
        self.not_empty.release()

def put(self, item, block=True, timeout=None):
    self.not_full.acquire()
    try:
        if self.maxsize > 0:
            if not block:
                if self._qsize() == self.maxsize:
                    raise Full
            elif timeout is None:
                while self._qsize() == self.maxsize:
                    self.not_full.wait()
            elif timeout < 0:
                raise ValueError("'timeout' must be a non-negative number")
            else:
                endtime = _time() + timeout
                while self._qsize() == self.maxsize:
                    remaining = endtime - _time()
                    if remaining <= 0.0:
                        raise Full
                    self.not_full.wait(remaining)
        self._put(item)
        self.unfinished_tasks += 1
        self.not_empty.notify()  # This is what wakes up `get`
    finally:
        self.not_full.release()

Problem

Original source