How to collect data from a list into groups based on condition?

grouping, parsing, python

Solution

You can use the fact that functions in Python have state. This grouper function serves the same purpose as DSM's `accumulate(fn(line) for line in s1)`:

def grouper(line):
    if is_event(line):
        grouper.count += 1
    return grouper.count
grouper.count = 0

result_as_iterators = (x[1] for x in itertools.groupby(lines, grouper))

Then if you need it:

result_as_lists = [list(x) for x in result_as_iterators]

To allow for concurrent use you need a new grouper function object each time you use it (so that it has its own count). You might find it simpler to make it a class:

class Grouper(object):
    def __init__(self):
        self.count = 0
    def __call__(self, line):
        if is_event(line):
            self.count += 1
        return self.count

results_as_iterators = itertools.groupby(lines, Grouper())

Problem

Not sure how to title this question. I've run into a few situations where I have a list of data, maybe annotated with some property, and I want to collect them into groups. For example, maybe I have a file like this: ``` some event reading: 25.4 reading: 23.4 reading: 25.1 different event reading: 22.3 reading: 21.1 reading: 26.0 reading: 25.2 another event reading: 25.5 reading: 25.1 ``` and I want to group each set of readings, splitting them on a condition (in this case, an event happening) so that I end up with a structure like ``` [['some event', 'reading: 25.4', 'reading: 23.4', 'reading: 25.1'], ['different event', 'reading: 22.3', 'reading: 21.1', 'reading: 26.0', 'reading: 25.2'], ['another event', 'reading: 25.5', 'reading: 25.1']] ``` In it's generic form, it is: Look for a condition, collect the data until that condition is true again, repeat Right now, I'd do something like ``` events = [] current_event = [] for line in lines: if is_event(line): if current_event: events.append(current_event) current_event = [line] else: current_event.append(line) else: if current_event: events.append(current_event) def is_event(line): return 'event' in line ``` which produces what I want, but it's ugly and hard to understand. I'm fairly certain there has to be a better way My guess is that it involves some itertools wizardry, but I'm new to itertools and can't quite wrap my head around all of it. Thanks! Update I've actually gone with Steve Jessop's answer with a Grouper class. Here's what I'm doing: ``` class Grouper(object): def __init__(self, condition_function): self.count = 0 self.condition_function = condition_function def __call__(self, line): if self.condition_function(line): self.count += 1 return self.count ``` and then using it like ``` event_grouper = Grouper(is_event) result_as_iterators = (x[1] for x in itertools.groupby(lines, event_grouper)) ``` and then to turn it into a dictionary I do ``` event_dictionary = [{event: readings} for event, *readings in result_as_iterators] ``` which gives ``` [ {'some event': ['reading: 25.4', 'reading: 23.4', 'reading: 25.1']}, {'different event': ['reading: 22.3','reading: 21.1','reading: 26.0','reading: 25.2']}, {'another event': ['reading: 25.5', 'reading: 25.1']} ] ```

Original source