processing text from a non-flat file (to extract information as if it *were* a flat file)

Question

I have a longitudinal data set generated by a computer simulation that can be represented by the following tables ('var' are variables): ``` time subject var1 var2 var3 t1 subjectA ... t2 subjectB ... ``` and ``` subject name subjectA nameA subjectB nameB ``` However, the file generated writes a data file in a format similar to the following: ``` time t1 description subjectA nameA var1 var2 var3 subjectB nameB var1 var2 var3 time t2 description subjectA nameA var1 var2 var3 subjectB nameB var1 var2 var3 ...(and so on) ``` I have been using a (python) script to process this output data into a flat text file so that I can import it into R, python, SQL, or awk/grep it to extract information - an example of the type of information desired from a single query (in SQL notation, after the data is converted to a table) is shown below: ``` SELECT var1, var2, var3 FROM datatable WHERE subject='subjectB' ``` I wonder if there is a more efficient solution as each of these data files can be ~100MB…

Accepted Answer

This is what Python generators are all about.

```
def read_as_flat( someFile ):
    line_iter= iter(someFile)
    time_header= None
    for line in line_iter:
        words = line.split()
        if words[0] == 'time':
            time_header = [ words[1:] ] # the "time" line
            description= line_iter.next()
            time_header.append( description )
        elif words[0] in subjectNameSet:
            data = line_iter.next()
            yield time_header + data
```

You can use this like a standard Python iterator

```
for time, description, var1, var2, var3 in read_as_flat( someFile ):
    etc.
```

processing text from a non-flat file (to extract information as if it were a flat file)

Solution

Problem