spray-can webservice graceful shutdown

akka, scala, spray, web-services

Solution

While I tried to use SIGTERM and JVM hook, I was in need to block the hook thread from exiting until my shutdown sequence finishes, and I simply don't know how to do it (I'm a bit inexperienced in akka, so maybe I missed some obvious solution).

What I finally did is attaching one more HTTP listener to localhost only, which has method to initiate shutdown (also it happened to be convenient for other tasks, like getting server status, triggering events in the application and such).

This can look like this (I suspect this may contain unnecessary actions, so refinements are welcome):

In bootstrap:

implicit val system = ActorSystem(...)

// create and start external service actor
val service = system.actorOf(Props[MyWebServiceActor], "my-web-service")

// create internal service to manage application
val controlService = system.actorOf(Props[ControlServiceActor], "control-service")

implicit val timeout = Timeout(10.seconds)

// start a new HTTP server for external service (notifying control of HTTP listener)
IO(Http).tell(Http.Bind(service, interface = config.interface, port = config.port), controlService)

// start internal server looking at localhost
IO(Http) ? Http.Bind(controlService, interface = "127.0.0.1", port = config.controlPort)

And the control service itself:

class ControlServiceActor extends Actor with HttpService with ActorLogging {
  def actorRefFactory = context
  implicit val system = context.system

  /* Listener of main service */
  var listener: ActorRef = _

  def receive = {
    // this is reply from IO.Http when HTTP listener is bound
    case Http.Bound(_) =>
      listener = sender()
      context.become(mainContext)
  }

  // http api for graceful stop
  val mainContext = runRoute {
    path("stop") {
      get {
        parameter('timeout.as[Int] ? 60) { timeout =>
          complete {
            // unbind makes listener to reject new connections
            context.become(shuttingDownContext)
            log.warning(s"Stopping application within $timeout seconds...")
            context.watch(listener)
            listener ! Http.Unbind(timeout.seconds)

            "Stopping..."
          }
        }
      }
    }
  }

  // Shutdown sequence
  val shuttingDownContext = ({
    // when unbound HTTP listener not accepting connections
    case Http.Unbound =>
      log.info("Webservice unbound, waiting for active connections to complete")

    // when HTTP listener terminated after unbound it has been processed all requests
    case Terminated(ref) if ref == listener =>
      log.info("Webservice finished, exiting")
      system.shutdown()
  }: Actor.Receive) orElse runRoute(complete("Shutdown in progress"))
}

Problem

I have spray.io based webservice, it runs as standalone jar (I use `sbt assembly` and then just `java -jar myws.jar`). It has pretty the same bootsrap as in spray examples, like this: ``` /** Bootstrap */ object Boot extends App { // we need an ActorSystem to host our application in implicit val system = ActorSystem("my-system") // create and start our service actor val service = system.actorOf(Props[MyServiceActor], "my-ws-service") implicit val timeout = Timeout(10.seconds) CLIOptionsParser.parse(args, CLIOptionsConfig()) map { config => // start a new HTTP server IO(Http) ? Http.Bind(service, interface = config.interface, port = config.port) } } ``` Now I just run the process in the backgroud with `java -jar my-service "$@" &` and stop with `kill -9 pid`. I'd like to stop my webservice gracefully, meaning that it finishes open connections and refuses new ones. Spray-can page on github recommends `to send it an Akka PoisonPill message`. Ideally I'd like to initiate it from command line, as simple as possible. I thought maybe to attach one more HTTP server instance bound to localhost only, and having some rest methods to stop, and maybe diagnose the webservice. Is it feasible? What are the other options? UPDATE: I added what I can imagine have to work, based on answers, but it seems not to, at least I've never seen any message I expected to see in stdout or log. Actually, I've tried in variations HttpUnbind, PoisonPill, together and by one. May anyone with a hard akka eye look at this? PS. The hook itself is called successfully, checked it. Signal I send to jvm is SIGTERM. ``` /* Simple reaper actor */ class Reaper(refs: ActorRef*) extends Actor { private val log = Logging(context.system, this) val watched = ArrayBuffer(refs: _*) refs foreach context.watch final def receive = { case Terminated(ref) => watched -= ref log.info(s"Terminated($ref)") println(s"Terminated($ref)") if (watched.isEmpty) { log.info("Shutting dow the system") println("Shutting dow the system") system.shutdown() } } } // termination hook to gracefully shutdown the service Runtime.getRuntime.addShutdownHook(new Thread() { override def run() = { val reaper = system.actorOf(Props(new Reaper(IO(Http), service))) //IO(Http) ? Http.Unbind(5.minutes) IO(Http) ! PoisonPill } }) ``` UPDATE2: So, somehow it works, namely - when PoisonPill is sent all current HTTP connections got closed. But I'd rather to stop receiveing new connections, and wait for open to return response and close. VERDICT: It seems that akka has its own hook, because, despite my hook gets executed, actors got killed and all connections got closed without my actions. If someone will offer solution with JVM shutdown hook it will be great. I suggest that this is important problem, and very sadly it has no any good recipe online. For a meanwhile I will try to implement graceful shutdown using tcp/http.

Original source