Difference between spring batch remote chunking and remote partitioning

java, spring, spring-batch

Solution

Remote Partitioning

Partitioning is a master/worker step configuration that allows for partitions of data to be processed in parallel. Each partition is described via some metadata. For example, if you were processing a database table, partition 1 may be ids 0-100, partition 2 being 101-200, etc. For Spring Batch, a master step uses a Partitioner to generate ExecutionContexts that contain the metadata for each partition. These ExecutionContexts are distributed to worker steps for processing by a PartitionHandler (for remote partitioning, the MessageChannelPartitionHandler is typically used). The workers execute their step and return the resulting statuses for aggregation by the master.

Things to note about remote partitioning:

- Input and output are local to the workers. For example, if the input is a file, the workers need access to the file.

- Workers need access to the JobRepository. Workers are fully defined Spring Batch steps and so they need JobRepository access.

Remote Chunking

Remote chunking is similar to remote partitioning in that it is a master/worker configuration. However with remote chunking, the data is read at by the master and sent over the wire to the worker for processing. Once the processing is done, the result of the ItemProcessor is returned to the master for writing.

Things to note about remote chunking:

- All I/O is done by the master.

- The workers handle processing only and therefore do not need JobRepository access.

- Remote chunking is more I/O intensive than remote partitioning since the actual data is sent over the wire instead of metadata describing it.

I did a talk on scaling Spring Batch and do a demonstration of remote partitioning that you can watch here: http://www.youtube.com/watch?v=CYTj5YT7CZU

Community Edit: In current documentation the term slave was replaced by worker, the answer has been edited to reflect that. Note some comments still reference the old term.

Problem

Original source