sparklyr: create new column with mutate function
apache-spark, r, sparklyr
Solution
I would strongly recommend you read the `sparklyr` documentation before proceeding. In particular, you're going to want to read the section on how R is translated to SQL (http://spark.rstudio.com/dplyr.html#sql_translation). In short, a very limited subset of R functions are available for use on `sparklyr` dataframes, and `gsub` is not one of those functions (but `toupper` is). If you really need `gsub` you're going to have to `collect` the data in to a local dataframe, then `gsub` it (you can still use `mutate`), then `copy_to` back to spark.
Problem
I'm very surprised if this kind of problems cannot be solved with sparklyr: ``` iris_tbl <- copy_to(sc, aDataFrame) # date_vector is a character vector of element # in this format: YYYY-MM-DD (year, month, day) for (d in date_vector) { ... aDataFrame %>% mutate(newValue=gsub("-","",d))) ... } ``` I receive this error: ``` Error: org.apache.spark.sql.AnalysisException: Undefined function: 'GSUB'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 2 pos 86 at org.apache.spark.sql.catalyst.catalog.SessionCatalog.failFunctionLookup(SessionCatalog.scala:787) at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:200) at org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:172) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:884) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:884) at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun ``` But with this line: ``` aDataFrame %>% mutate(newValue=toupper("hello")) ``` things work. Some help?