How to rename huge amount of files in Hadoop/Spark?
apache-spark, bigdata, hadoop, parallel-processing
Solution
The problem is that you are trying to serialize the ghfs object. If you use mapPartitions and recreate the ghfs object in each partition you will be able to run your code with just a couple of minor changes.
Problem
I have an input folder that contains +100,000 files. I would like to do a batch operation on them, i.e. rename all of them in a certain way, or move them to a new path based on information in each file's name. I would like to use Spark to do that, but unfortunately when I tried the following piece of code: ``` final org.apache.hadoop.fs.FileSystem ghfs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI(args[0]), new org.apache.hadoop.conf.Configuration()); org.apache.hadoop.fs.FileStatus[] paths = ghfs.listStatus(new org.apache.hadoop.fs.Path(args[0])); List<String> pathsList = new ArrayList<>(); for (FileStatus path : paths) { pathsList.add(path.getPath().toString()); } JavaRDD<String> rddPaths = sc.parallelize(pathsList); rddPaths.foreach(new VoidFunction<String>() { @Override public void call(String path) throws Exception { Path origPath = new Path(path); Path newPath = new Path(path.replace("taboola","customer")); ghfs.rename(origPath,newPath); } }); ``` I get an error that hadoop.fs.FileSystem is not Serializable (and therefore probably cannot be used in parallel operations) Any idea how I can workaround it or have it done another way?