How can I parallize the execution of my plugins with Celluloid?

celluloid, concurrency, multithreading, ruby

Solution

You just need a supervisor if your plugins might crash and you want celluloid to restart them.

What you want to do could be as simple as using pools and futures.

To have a shared pool for your plugins you need a new actor for that.

class PluginRunner
  include Celluloid

  def run(plugin)
    plugin.run
  end
end

plugin_runner = PluginRunner.pool(size: 4)

plugins = [LastFmPlugin, TwitterPlugin]
results = plugins.map {|p| plugin_runner.future.run(p) }.map(&:value)
persist_results(results)

Problem

My question should help me to get on the right way. I'm developing a ruby application with the concurrent framework `celluloid`. Here how it looks like: I have some plugins. I want to run them concurrently and wait until the last one has finished. I've an abstract class, called `PluginFrame`, which is inherited by all the plugin and which also provides a `run` method. - My idea was to make a `SupervisionGroup`, but is that the right idea? - How can I run a `SupervisionGroup` and wait until all group members have finished? - It's a good idea to make a separate `PluginPool` class, to manage the pool of plugins? - It's also interesting for me to limit the `pool` size, so only two or three plugins run at the same time.How can I achieve this?

Original source