Optimise filtering lists in Python 2.7

generator, python, python-2.7

Solution

First of all using `filter`/`lambda` combination is going to be deprecated. Current functional programming style is described in Python Functional Programming HOWTO.

Secondly, if you concerned with efficiency, rather than construct lists, you should return generators. In this case they are simple enough to use generator expressions.

def get_clothes():
    return (t for t in allThings if t.garment)

def get_hats():
    return (t for t in get_clothes() if t.headgear)

Or if you'd prefer, true generators (allegedly more pythonic):

def get_clothes():
    for t in allThings:
       if t.garment:
           yield t

def get_hats():
    for t in get_clothes():
        if t.headgear:
            yield t

If for some reason, sometimes you need `list` rather than `iterator`, you can construct list by simple casting:

hats_list = list(get_hats())

Note, that above will not construct list of clothes, thus efficiency is close to your duplicate code version.

Problem

I need to filter a large lists several times, but I'm concerned with both simplicity of code and execution efficiency. To give an example: ``` all_things # huge collection of all things # inefficient but clean code def get_clothes(): return filter(lambda t: t.garment, allThings) def get_hats(): return filter(lambda t: t.headgear, get_clothes()) ``` I'm concerned that I'm iterating over the clothes list when in fact it has already been iterated over. I also want to keep the two filter operations separate, as they belong to two different classes, and I do not want to duplicate the first lambda function in the hats class. ``` # efficient but duplication of code def get_clothes(): return filter(lambda t: t.garment, allThings) def get_hats(): return filter(lambda t: t.headgear and t.garment, allThings) ``` I have been investigating generator functions, as they seemed like the way to go, but I haven't as yet figure out how.

Original source