how to use Regexp_replace in spark

apache-spark, apache-spark-sql, regexp-replace, scala

Solution

Here's a reproducible example, assuming `x4` is a string column.

import org.apache.spark.sql.functions.regexp_replace

val df = spark.createDataFrame(Seq(
  (1, "1,3435"),
  (2, "1,6566"),
  (3, "-0,34435"))).toDF("Id", "x4")

The syntax is `regexp_replace(str, pattern, replacement)`, which translates to:

df.withColumn("x4New", regexp_replace(df("x4"), "\\,", ".")).show
+---+--------+--------+
| Id|      x4|   x4New|
+---+--------+--------+
|  1|  1,3435|  1.3435|
|  2|  1,6566|  1.6566|
|  3|-0,34435|-0.34435|
+---+--------+--------+

Problem

I am pretty new to spark and would like to perform an operation on a column of a dataframe so as to replace all the `,` in the column with `.` Assume there is a dataframe x and column x4 ``` x4 1,3435 1,6566 -0,34435 ``` I want the output to be as ``` x4 1.3435 1.6566 -0.34435 ``` The code I am using is ``` import org.apache.spark.sql.Column def replace = regexp_replace((x.x4,1,6566:String,1.6566:String)x.x4) ``` But I get the following error ``` import org.apache.spark.sql.Column <console>:1: error: ')' expected but '.' found. def replace = regexp_replace((train_df.x37,0,160430299:String,0.160430299:String)train_df.x37) ``` Any help on the syntax, logic or any other suitable way would be much appreciated

Original source