AWS DynamoDB and MapReduce in Java

Question

I have a huge DynamoDB table that I want to analyze to aggregate data that is stored in its attributes. The aggregated data should then be processed by a Java application. While I understand the really basic concepts behind MapReduce, I've never used it before. In my case, let's say that I have a `customerId` and `orderNumbers` attribute in every DynamoDB item, and that I can have more than one item for the same customer. Like: ``` customerId: 1, orderNumbers: 2 customerId: 1, orderNumbers: 6 customerId: 2, orderNumbers: -1 ``` Basically I want to sum the orderNumbers for each customerId, and then execute some operations in Java with the aggregate. AWS Elastic MapReduce could probably help me, but I don't understand how do I connect a custom JAR with DynamoDB. My custom JAR probably needs to expose both a `map` and `reduce` functions, where can I find the right interface to implement? Plus I'm a bit confused by the docs, it seems like I should first export my data to S3 before running…

Accepted Answer

Note: I haven't built a working EMR, just read about it.

First of all, Prerequisites for Integrating Amazon EMR with Amazon DynamoDB

You can work directly on DynamoDB: Hive Command Examples for Exporting, Importing, and Querying Data in Amazon DynamoDB, As you can see you can do "SQL-like" queries that way.

If you have zero knowledge about Hadoop you should probably read some introduction material such as: What is Hadoop

This tutorial is another good read Using Amazon Elastic MapReduce with DynamoDB

Regarding your custom JAR application, you need to upload it to S3. Use this guide: How to Create a Job Flow Using a Custom JAR

I hope this will help you get started.

AWS DynamoDB and MapReduce in Java

Solution

Problem