How to execute a for loop in batches?

batch-processing, python, python-2.7

Solution

The general structure I use looks like this:

worklist = [...]
batchsize = 500

for i in range(0, len(worklist), batchsize):
    batch = worklist[i:i+batchsize] # the result might be shorter than batchsize at the end
    # do stuff with batch

Note that we're using the `step` argument of `range` to simplify the batch processing considerably.

In Python 3.12, you can now use `itertools.batched` to achieve the same result:

from itertools import batched

for batch in batched(worklist, batchsize):
    # process the `batch` tuple

Problem

``` for x in records: data = {} for y in sObjectName.describe()['fields'] data[y['name']] = x[y['name']] ls.append(adapter.insert_posts(collection, data)) ``` I want to execute the code ls.append(adapter.insert_post(collection, x)) in the batch size of 500, where x should contain 500 data dicts. I could create a list a of 500 data dicts using a double for loop and a list and then insert it. I could do that in the following way, , is there a better way to do it? : ``` for x in records: for i in xrange(0,len(records)/500): for j in xrange(0,500): l=[] data = {} for y in sObjectName.describe()['fields']: data[y['name']] = x[y['name']] #print data #print data l.append(data) ls.append(adapter.insert_posts(collection, data)) for i in xrange(0,len(records)%500): l=[] data = {} for y in sObjectName.describe()['fields']: data[y['name']] = x[y['name']] #print data #print data l.append(data) ls.append(adapter.insert_posts(collection, data)) ```

Original source