Why not load balance when parallel computing using snowfall?

Question

For a long time I have been using sfLapply for a lot of my parallel r scripts. However, recently as I have delved more into parallel computing, I have been using sfClusterApplyLB, which can save a lot of time if individual instances do not take the same amount of time to run. Were as sfLapply will wait for each instance of a batch to finish before loading a new batch (which may lead to idle instances), with sfClusterApplyLB instances that complete their task will immediately be assigned to remaining elements in the list thus potentially saving quite a bit of time when instances do not take precisely the same amount of time. This has led me to question why would we ever want to NOT load balance our runs when using snowfall? The only thing I have found so far is that, when there is an error in the paralleled script, sfClusterApplyLB will still cycle through the entire list before giving an error, while sfLapply will stop after trying the first batch. What else am I missing? are there an…

Accepted Answer

The `sfLapply` function is useful because it splits up the input values into one group of tasks for each available worker, which is what the `mclapply` function calls prescheduling. This can give much better performance than `sfClusterApplyLB` when the tasks don't take long.

Here's an extreme example that demonstrates the advantages of prescheduling:

```
> system.time(sfLapply(1:100000, sqrt))
   user  system elapsed
  0.148   0.004   0.170
> system.time(sfClusterApplyLB(1:100000, sqrt))
   user  system elapsed
 19.317   1.852  21.222
```

Why not load balance when parallel computing using snowfall?

Solution

Problem