Maven dependencies for Hadoop: MiniDFSCluster & MiniMRCluster

hadoop, hadoop2, unit-testing

Solution

Guess I figured it out. In your Maven pom file, first add a new repository:

<repositories>
    <repository>
        <id>cloudera</id>
        <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
    </repository>
</repositories>

Then add the following to your project dependencies

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.1</version>
</dependency>
<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.11</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-auth</artifactId>
    <version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-test</artifactId>
    <version>2.0.0-mr1-cdh4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.0.0-cdh4.3.0</version>
    <classifier>tests</classifier>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.0.0-cdh4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.0.0-cdh4.3.0</version>
    <classifier>tests</classifier>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.0.0-mr1-cdh4.3.0</version>
</dependency>

In case someone is interested to get the whole project (unit test for the famous WordCount MapReduce job, I am willing to share it)

Problem

I want to implement a maven project, that helps me unit test a Hadoop MapReduce job. My biggest problem is defining the Maven dependencies to be able to make use of the test classes: MiniDFSCluster & MiniMRCluster. I am using Hadoop 2.4.1. Any ideas?

Original source