Is there any means to serialize custom Transformer in Spark ML Pipeline

apache-spark, apache-spark-mllib, apache-spark-sql, serialization

Solution

If you are using Spark 2.x+ then extend your transformer with DefaultParamsWritable

for example

class ProbabilityMaxer extends Transformer with DefaultParamsWritable{

Then create a constructor with a string parameter

 def this(_uid: String) {
    this()
  }

Finally for a successful read add a companion class

object ProbabilityMaxer extends  DefaultParamsReadable[ProbabilityMaxer]

I have this working on my production server. I will add gitlab link to the project later when I upload it

Problem

I use ML pipeline with various custom UDF-based transformers. What I'm looking for is a way to serialize/deserialize this pipeline. I serialize the PipelineModel using ``` ObjectOutputStream.write() ``` However whenever I try to deserialize the pipeline I'm having: ``` java.lang.ClassNotFoundException: org.sparkexample.DateTransformer ``` Where is DateTransformer is my custom transformer. Is there any method/interface to implement for proper serialization? I've found out there is ``` MLWritable ``` Interface that might be implemented by my class (DateTransformer extends Transfrormer), however can't find useful example of it.

Original source