Accumulator fails on cluster, works locally

Question

In the official spark documentation, there is an example for an accumulator which is used in a `foreach` call which is directly on an RDD: ``` scala> val accum = sc.accumulator(0) accum: spark.Accumulator[Int] = 0 scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x) ... 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s scala> accum.value res2: Int = 10 ``` I implemented my own accumulator: ``` val myCounter = sc.accumulator(0) val myRDD = sc.textFile(inputpath) // :spark.RDD[String] myRDD.flatMap(line => foo(line)) // line 69 def foo(line: String) = { myCounter += 1 // line 82 throwing NullPointerException // compute something on the input } println(myCounter.value) ``` In a local setting, this works just fine. However, if I run this job on a spark standalone cluster with several machines, the workers throw a ``` 13/07/22 21:56:09 ERROR executor.Executor: Exception in task ID 247 java.lang.NullPointerException at MyClass$.foo(MyClass.scala:82) at MyClass$$…

Accepted Answer

In my case too, accumulator was null in closure when I used 'extends App' to create a spark application as shown below

```
    object AccTest extends App {

val conf = new SparkConf().setAppName("AccTest").setMaster("yarn-client")
    val sc = new SparkContext(conf)
    sc.setLogLevel("ERROR")

val accum = sc.accumulator(0, "My Accumulator")
    sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)

println("count:" + accum.value)

sc.stop
  }
}
```

I replaced extends App with main() method and it worked in YARN cluster in HDP 2.4

```
object AccTest {

def main(args: Array[String]): Unit = {

val conf = new SparkConf().setAppName("AccTest").setMaster("yarn-client")
        val sc = new SparkContext(conf)
        sc.setLogLevel("ERROR")

val accum = sc.accumulator(0, "My Accumulator")
        sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)

println("count:" + accum.value)

sc.stop
    }
}
```

worked

Accumulator fails on cluster, works locally

Solution

Problem