Return multiple items from spring batch ItemProcessor

batch-processing, itemprocessor, java, spring, spring-batch

Solution

Item Processor takes one thing in, and returns a list

MyItemProcessor implements ItemProcessor<SingleThing,List<ExtractedThingFromSingleThing>> {
    public List<ExtractedThingFromSingleThing> process(SingleThing thing) {
    //parse and convert to list
    }
}

Wrap the downstream writer to iron things out. This way stuff downstream from this writer doesn't have to work with lists.

@StepScope
public class ItemListWriter<T> implements ItemWriter<List<T>> {
    private ItemWriter<T> wrapped;

    public ItemListWriter(ItemWriter<T> wrapped) {
        this.wrapped = wrapped;
    }

    @Override
    public void write(List<? extends List<T>> items) throws Exception {
        for (List<T> subList : items) {
            wrapped.write(subList);
        }
    }
}

Problem

I'm writing a spring batch job and in one of my step I have the following code for the processor: ``` @Component public class SubscriberProcessor implements ItemProcessor<NewsletterSubscriber, Account>, InitializingBean { @Autowired private AccountService service; @Override public Account process(NewsletterSubscriber item) throws Exception { if (!Strings.isNullOrEmpty(item.getId())) { return service.getAccount(item.getId()); } // search with email address List<Account> accounts = service.findByEmail(item.getEmail()); checkState(accounts.size() <= 1, "Found more than one account with email %s", item.getEmail()); return accounts.isEmpty() ? null : accounts.get(0); } @Override public void afterPropertiesSet() throws Exception { Assert.notNull(service, "account service must be set"); } } ``` The above code works but I've found out that there are some edge cases where having more than one `Account` per `NewsletterSubscriber` is allowed. So I need to remove the state check and to pass more than one `Account` to the item writer. One solution I found is to change both `ItemProcessor` and `ItemWriter` to deal with `List<Account>` type instead of `Account` but this have two drawbacks: - Code and tests are uglier and harder to write and maintain because of nested lists in writer - Most important more than one `Account` object may be written in the same transaction because a list given to writer may contain multiple accounts and I'd like to avoid this. Is there any way, maybe using a listener, or replacing some internal component used by spring batch to avoid lists in processor? Update I've opened an issue on spring Jira for this problem. I'm looking into isComplete and getAdjustedOutputs methods in `FaultTolerantChunkProcessor` which are marked as extension points in `SimpleChunkProcessor` to see if I can use them in some way to achieve my goal. Any hint is welcome.

Original source