Discarding FIFO queue, with max capacity, in Python?

fifo, python, python-2.7, queue

Solution

Use `collections.deque` specifying a `maxlen`,

>>> q = deque(maxlen=2)
>>> q.extend([1, 2, 3])
>>> q
deque([2, 3], maxlen=2)

Problem

I need a discarding FIFO queue, that automatically discards items when it becomes full. It doesn't have to be thread-safe. Efficiency if more important. That is, I need to sample signals from a device while being able to retrieve the last n seconds (objects) at semi-random times. I have implemented a (not so thread-safe) buffer myself, but have the feeling I'm re-inventing the wheel here. In addition we are talking about a 100 objects every second. Most will be discarded while e.g. 3000 (= 30 secs of data) will have to be retrieved at a time (say every ten minutes). Is there such a class already in Python standard libs or elsewhere? I've used some goggle-fu but not managed to find anything useful. DiscardingBuffer ``` from Queue import Queue, Full, Empty import logging class DiscardingBuffer(): def __init__(self, capacity=0): self._queue = Queue(maxsize=capacity) def put(self, item): while True: try: self._queue.put(item, block=False) logging.debug('Put item: {0}'.format(item)) break except Full: discarded_item = self._queue.get(block=False) logging.debug('Buffer is full. Discarding: {0}'.format(discarded_item)) def flush(self): items = [] while True: try: items.append(self._queue.get(block=False)) except Empty: logging.debug('Buffer is now empty.') break return items def main(): buf = DiscardingBuffer(5) for i in xrange(10): buf.put(i) logging.debug('Remaining items: {0}'.format(buf.flush())) logging.debug('Verify it is empty: {0}'.format(buf.flush())) if __name__ == '__main__': logging.basicConfig(level=logging.DEBUG, format='[%(levelname)1.1s %(asctime)s %(name)s (%(process)d):%(lineno)d] %(message)s', datefmt='%Y-%m-%d %H:%M:%S') main() ``` Output ``` [D 2013-08-22 10:13:58 root (4164):13] Put item: 0 [D 2013-08-22 10:13:58 root (4164):13] Put item: 1 [D 2013-08-22 10:13:58 root (4164):13] Put item: 2 [D 2013-08-22 10:13:58 root (4164):13] Put item: 3 [D 2013-08-22 10:13:58 root (4164):13] Put item: 4 [D 2013-08-22 10:13:58 root (4164):17] Buffer is full. Discarding: 0 [D 2013-08-22 10:13:58 root (4164):13] Put item: 5 [D 2013-08-22 10:13:58 root (4164):17] Buffer is full. Discarding: 1 [D 2013-08-22 10:13:58 root (4164):13] Put item: 6 [D 2013-08-22 10:13:58 root (4164):17] Buffer is full. Discarding: 2 [D 2013-08-22 10:13:58 root (4164):13] Put item: 7 [D 2013-08-22 10:13:58 root (4164):17] Buffer is full. Discarding: 3 [D 2013-08-22 10:13:58 root (4164):13] Put item: 8 [D 2013-08-22 10:13:58 root (4164):17] Buffer is full. Discarding: 4 [D 2013-08-22 10:13:58 root (4164):13] Put item: 9 [D 2013-08-22 10:13:58 root (4164):26] Buffer is now empty. [D 2013-08-22 10:13:58 root (4164):38] Remaining items: [5, 6, 7, 8, 9] [D 2013-08-22 10:13:58 root (4164):26] Buffer is now empty. [D 2013-08-22 10:13:58 root (4164):39] Verify it is empty: [] ```

Original source