How to parallelize an RDD?
apache-spark, scala
Solution
Resilient Distributed Datasets (RDDs) RDD as the name suggests are distributed and fault-tolerant and parallel.
"RDDs are fault-tolerant, parallel data structures that let users explicitly persist intermediate results in memory, control their partitioning to optimize data placement, and ma- nipulate them using a rich set of operators." Please see this paper.
No you don't need to convert an RDD to a Seq object. All processing on RDDs are done in parallel (depending on how parallel your Spark installation is).
Problem
To read a file into memory I use : ``` val lines = sc.textFile("myLogFile*") ``` which is of type : ``` org.apache.spark.rdd.RDD[String] = MappedRDD[1] at textFile at <console>:12 ``` Reading the Scala doc at : http://spark.apache.org/docs/0.9.1/scala-programming-guide.html#parallelized-collections "Parallelized collections are created by calling SparkContext’s parallelize method on an existing Scala collection (a Seq object)." This does not seem to apply to an RDD ? Can parallelized processing occur on an RDD ? Do I need to convert the RDD to a Seq object ?