Parallel version of Files.walkFileTree (java or scala)

file-processing, io, java, multithreading, scala

Solution

Let's assume that executing a callback on each file is enough.

This code will not handle loops in the file system--you'd need a registry of where you've been for that (e.g. `java.util.concurrent.ConcurrentHashMap`). There are all sorts of improvements you could add, like reporting exceptions instead of silently ignoring them.

import java.io.File
import scala.util._
def walk(f: File, callback: File => Unit, pick: File => Boolean = _ => true) {
  Try {
    val (dirs, fs) = f.listFiles.partition(_.isDirectory)
    fs.filter(pick).foreach(callback)
    dirs.par.foreach(f => walk(f, callback, pick))
  }
}

Collecting the files using a fold instead of a `foreach` is not drastically harder, but I leave that as an exercise to the reader. (A `ConcurrentLinkedQueue` is probably fast enough to accept them all in a callback anyway unless you have really slow threads and a awesome filesystem.)

Problem

Original source