R parallel S4 class cluster error

parallel-processing, r, r-s4

Solution

So here's what's going on:

For S4 objects of class "Animal", the `count` function simply extracts the `legs` slot. If this were all that you were doing, you wouldn't need to evaluate or source the file `animal.R` on your cluster nodes. All necessary information would be passed by `parSapply`.

However, the `countAfterChopping` function assigns a new value to the `legs` slot, and this is where the fun begins. The slot assignment function ``@<-`` contains a call to ``slot<-`` with the argument `check = TRUE`. This triggers an evaluation of the function `checkSlotAssignment`, which checks "that the value provided is allowed for this slot, by consulting the definition of the class" (from `?checkSlotAssignment`).

Therefore, the class definition must be known when assigning to a slot in this way, and the S4 class "Animal" is not known on the cluster nodes. This is why evaluating the parsed file `animal.R` or sourcing it works. However, you would be fine with just evaluating the first line of the file, i.e., defining the class "Animal", on each node.

Here's a reduced, reproducible example:

animal.R<-"
  setClass('Animal', representation(species = 'character', legs = 'numeric'))

  ##Define some Animal methods
  setGeneric('count',function(x) standardGeneric('count'))
  setMethod('count', signature(x='Animal'), function(x) { x@legs})

  setGeneric('countAfterChopping',function(x) standardGeneric('countAfterChopping'))
  setMethod('countAfterChopping', signature(x='Animal'),
    function(x) { x@legs <- x@legs-1; x@legs})
"
library(parallel)

source(textConnection(animal.R))

cl <- makeCluster(rep('localhost', 2))

daisy<-new("Animal",legs=2,species="H.sapiens")
fred<-new("Animal",legs=4,species="C.lupus")

parSapply(cl, list(daisy, fred), count)
# [1] 2 4

clusterExport(cl,"animal.R") # 
clusterEvalQ(cl,eval(parse(textConnection(animal.R),n=1)))

parSapply(cl, list(daisy, fred), countAfterChopping)
# [1] 1 3

Problem

Original source