python - creating dictionary from comma-separated lines, containing nested values
dictionary, input, parsing, python
Solution
If and only if there is not nested bracket it's a perfect fit for regexp.
import re
line = 'A=15, B=8, C=false, D=[somevar, a=0.1, b=77, c=true]'
regexp = r'(\w*)=(\[[^\]]*\]|[^,]*),?\s*'
print(dict((match.group(1),match.group(2)) for match in re.finditer(regexp,line)))
output
{'A': '15', 'C': 'false', 'B': '8', 'D': '[somevar, a=0.1, b=77, c=true]'}
concerning your fear of being not fast enought, don't assume mesure. As the regexp is optimised C (except for few pathological cases) there is few chance you can do better.
Problem
I have got a line in such format: ``` line = 'A=15, B=8, C=false, D=[somevar, a=0.1, b=77, c=true]' ``` I would like to extract these values to a dictionary, getting such result: ``` { 'A': '15', 'B': '8', 'C': 'false', 'D': '[somevar, a=0.1, b=77, c=true]' } ``` If not the D value, I could have used such simple method: ``` result = dict(e.split('=') for e in line.split(', ')) ``` But given the fact that D contains `', '` as a separator, I get a total mess instead ``` { 'A': '15', 'B': '8', 'C': 'false', 'D': '[somevar', 'a': '0.1', 'b': '77', 'c': 'true]' } ``` I would appreciate any advice - I have not tried with regexps yet, but this thing has to be fast, as there are dozens of gigabytes of such lines, and I am afraid that regexping would slow it down a lot... EDIT: Benchmarks I have wrapped most of the answers below into functions and used ipython's `%timeit` magic function to measure the execution times. The test file was created on tmpfs in RAM by doing simply: ``` for i in {1..1000000}; do echo 'A=15, B=8, C=false, D=[somevar, a=0.1, b=77, c=true]' >> test_file; done ``` This is how the complete test program looked like: ``` import shlex import re def kalgasnik(line): lexer = shlex.shlex(line) lexer.wordchars += '.' values = [['']] stack = [values] for token in lexer: if token == ',': stack[-1] += [['']] elif token == '=': stack[-1][-1] += [''] elif token == '[': v = [['']] stack[-1][-1][-1] = v stack += [v] elif token == ']': sub = stack.pop() stack[-1][-1][-1] = {v[0]: v[1] if len(v) > 1 else None for v in sub} else: stack[-1][-1][-1] += token values = {v[0]: v[1] if len(v) > 1 else None for v in values} return values def roberto(myline): mydict = {} parsecheck = {'(':1, '[':1, '{':1, ')':-1, ']':-1, '}':-1} parsecount = 0 chargroup = '' myline = myline + ',' for thischar in myline: parsecount += parsecheck.get(thischar, 0) if parsecount == 0: if thischar == '=': thiskey = chargroup.strip() chargroup = '' elif thischar == ',': mydict[thiskey] = chargroup chargroup = '' else: chargroup += thischar else: chargroup += thischar return mydict def xavier(line): regexp = r'(\w*)=(\[[^\]]*\]|[^,]*),?\s*' outdict = dict((match.group(1),match.group(2)) for match in re.finditer(regexp,line)) return outdict def wim(line): outdict = dict(x.split('=', 1) for x in shlex.split(line.replace("[", "'[").replace("]", "]'"))) return outdict def gorkypl(line): outdict = dict(e.split('=') for e in line.split(', ')) return outdict def run_test(method): with open('test_file', 'r') as infile: for line in infile: method(line) ``` And here are the results: ``` %timeit run_test(kalgasnik) 1 loops, best of 3: 3min 52s per loop %timeit run_test(roberto) 1 loops, best of 3: 30.2 s per loop %timeit run_test(xavier) 1 loops, best of 3: 12.1 s per loop %timeit run_test(wim) 1 loops, best of 3: 2min 41 s per loop ``` And for the sake of comparison, the (not-working-correctly) original idea based purely on `split`. ``` %timeit run_test(gorkypl) 1 loops, best of 3: 8.27 s per loop ``` So, basically, Xavier's regexp-based solution is not only the most flexible, but the fastest one, and not that much slower than the naive method based on `split()`. Thank you all a lot!