Persisting Spark Dataframe

Question

I am new to the Spark world. How we can persist a Dataframe so that we can use it across the components. I have a Kafka stream from which, I am producing the Dataframe through the Rdd.Tried RegisterAsTempTable ,but the table is not accessible in another program. I want to access this Dataframe in another class through sqlContext and use the query result for further calculations.

Accepted Answer

You can save the contents of a DataFrame as a Parquet file and read the same in another program. you can register as Temp table in next program.Spark SQL provides support for both reading and writing Parquet files that automatically preserves the schema of the original data.

```
//First Program
dataframe.write.format("parquet").save("/tmp/xyz-dir/card.parquet")
//where /tmp/xyz-dir/ is a HDFS directory

//Second Program
val parquetRead = sqlContext.read.format("parquet").load("/tmp/xyz-dir/card.parquet")

//Parquet files can also be registered as tables and then used in SQL statements.
parquetRead.registerTempTable("parquettemptable")
val cust= sqlContext.sql("SELECT name FROM parquettemptable")

//After use of parquet file, delete the same in the second program
val fs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI("hdfs://hostname:8030"), sc.hadoopConfiguration)
fs.delete(new org.apache.hadoop.fs.Path("/tmp/xyz-dir"),true) // isRecusrive= true
```

Persisting Spark Dataframe

Solution

Problem