Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
apache-spark, apache-spark-dataset, java, rdd, scala
Solution
`sparkContext` is a Scala implementation entry point and `JavaSparkContext` is a java wrapper of `sparkContext`.
`SQLContext` is entry point of SparkSQL which can be received from `sparkContext`.Prior to 2.x.x, RDD ,DataFrame and Data-set were three different data abstractions.Since Spark 2.x.x, All three data abstractions are unified and `SparkSession` is the unified entry point of Spark.
An additional note is , RDD meant for unstructured data, strongly typed data and DataFrames are for structured and loosely typed data. You can check
Is there any method to convert or create Context using Sparksession ?
yes. its `sparkSession.sparkContext()` and for SQL, `sparkSession.sqlContext()`
Can I completely replace all the Context using one single entry SparkSession ?
yes. you can get respective contexs from sparkSession.
Does all the functions in SQLContext, SparkContext,JavaSparkContext etc are added in SparkSession?
Not directly. you got to get respective context and make use of it.something like backward compatibility
How to use such function in SparkSession?
get respective context and make use of it.
How to create the following using SparkSession?
- RDD can be created from `sparkSession.sparkContext.parallelize(???)`
- JavaRDD same applies with this but in java implementation
- JavaPairRDD `sparkSession.sparkContext.parallelize(???).map(//making your data as key-value pair here is one way)`
- Dataset what sparkSession returns is Dataset if it is structured data.
Problem
- What is the difference between `SparkContext,` `JavaSparkContext,` `SQLContext` and `SparkSession`? - Is there any method to convert or create a Context using a `SparkSession`? - Can I completely replace all the Contexts using one single entry `SparkSession`? - Are all the functions in `SQLContext`, `SparkContext`, and `JavaSparkContext` also in `SparkSession`? - Some functions like `parallelize` have different behaviors in `SparkContext` and `JavaSparkContext`. How do they behave in `SparkSession`? How can I create the following using a `SparkSession`? - `RDD` - `JavaRDD` - `JavaPairRDD` - `Dataset` Is there a method to transform a `JavaPairRDD` into a `Dataset` or a `Dataset` into a `JavaPairRDD`?