Why Spark SQL translates String "null" to Object null for Float/Double types?
apache-spark, apache-spark-sql, scala
Solution
If you looked in to the `fill` function in `Dataset`, It checks the datatype and tries to convert to datatype of its column's schema. If it can be converted then it converts otherwise it returns null.
It does not convert to "`null`" to object `null` but it returns null if exception occurs while converting.
val map = df.columns.map((_, "WHATEVER")).toMap
gives null
and val map = df.columns.map((_, "9999.99")).toMap
gives 9999.99
If you want to update the `NAN` with same datatype, you can get result as expected.
Hope this helps you to understand!
Problem
I have a dataframe containing `float` and `double` values. ``` scala> val df = List((Float.NaN, Double.NaN), (1f, 0d)).toDF("x", "y") df: org.apache.spark.sql.DataFrame = [x: float, y: double] scala> df.show +---+---+ | x| y| +---+---+ |NaN|NaN| |1.0|0.0| +---+---+ scala> df.printSchema root |-- x: float (nullable = false) |-- y: double (nullable = false) ``` When I replace `NaN` values with `null` value, I gave `null` as String to the Map in `fill` operation. ``` scala> val map = df.columns.map((_, "null")).toMap map: scala.collection.immutable.Map[String,String] = Map(x -> null, y -> null) scala> df.na.fill(map).printSchema root |-- x: float (nullable = true) |-- y: double (nullable = true) scala> df.na.fill(map).show +----+----+ | x| y| +----+----+ |null|null| | 1.0| 0.0| +----+----+ ``` And I got correct value. But I was not able to understand as to How/Why Spark SQL is translating `null` as a String to a `null` object ?