What's the difference between spark-shell and spark-sql? Any performance differences?
apache-spark, apache-spark-sql
Solution
`spark-shell` gives you a working Spark environment where Scala is the (programming) language.
`spark-sql` gives you a Spark SQL environment where SQL is the query language.
Note that `spark-shell` is for any APIs available in Spark while `spark-sql` is only for Spark SQL API (with Datasets and DataFrames).
They're simply different interfaces for users with different skills (`spark-shell` for Spark/Scala developers while `spark-sql` for SQL developers).
`spark-sql` "hides" the Spark infrastructure behind SQL interface which places it higher in how much engineering skills one should have, but eventually uses all the optimizations available in Spark SQL (and Spark in general).
Performance-wise `spark-sql` and `spark-shell` are alike.
Problem
`Spark-shell`: which basically opens the `scala>` prompt. Where query needs to write in below manner ``` val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) // Queries are expressed in HiveQL sqlContext.sql("FROM src SELECT key, value").collect().foreach(println) ``` `spark-sql`: Which seems to connect directly to hive metastore and we can write a query in a similar way to hive. And query over existing data in hive I want to know the difference between these two.. And does processing any query in spark-sql is same as in spark-shell? I mean can we leverage the performance benefits of spark in spark-sql? Spark 1.5.2's here.