Misk Service Lifecycle: Readiness and Graceful Shutdown Done Right


Series: Building Production Services with Misk — Part 4 of 24

Binding a service into the graph is the easy part. The hard part — the part that decides whether a deploy drops requests or not — is lifecycle: what starts before what, when the container reports itself ready, and whether it finishes in-flight work before it dies. The misk service lifecycle is one of the framework’s genuinely strong, genuinely under-documented areas, and getting it right is most of what separates a toy from something you’d page on. Let’s follow a service from startAsync to SIGTERM.

Services are Guava Services

Long-running work in Misk — database connections, the HTTP server, a queue consumer — is modeled as a Guava Service: an object with a managed lifecycle that moves through states NEW → STARTING → RUNNING → STOPPING → TERMINATED. Misk doesn’t reinvent this; it leans on Guava’s ServiceManager to coordinate a whole set of them.

You rarely implement Service from scratch. As the docs advise, inherit one of Guava’s base classes — AbstractIdleService (start/stop hooks, no loop), AbstractScheduledService (runs on a schedule), or AbstractExecutionThreadService (owns a thread). You bind a service by installing a ServiceModule:

class MyServiceModule : KAbstractModule() {
  override fun configure() {
    install(ServiceModule<MovieService>())
  }
}

MiskApplication collects every service installed this way and hands them to the ServiceManager, which starts them, waits for all to report healthy, and stops them on shutdown. So far, so ordinary. The interesting part is ordering.

Declaring order: dependsOn and enhancedBy

Real services have a startup order. Your MovieService is useless until its DatabaseService is up. Misk lets you declare that at the install site, and the ServiceManager enforces it:

install(
  ServiceModule<MovieService>()
    .dependsOn<DatabaseService>()
)

The guarantee is symmetric and precise, straight from the source: MovieService does not enter STARTING until DatabaseService is RUNNING, and on the way down MovieService must reach TERMINATED before DatabaseService enters STOPPING. Dependencies have dependencies, so this forms a graph the manager topologically orders.

There’s a second, subtler relationship: enhancement. Some services exist only to augment another — a SchemaMigrationService that creates tables for a DatabaseService, say. That’s an implementation detail of the database, and the rest of your app shouldn’t have to know it exists. So you express it as an enhancement:

install(ServiceModule<SchemaMigrationService>())
install(
  ServiceModule<DatabaseService>()
    .enhancedBy<SchemaMigrationService>()
)
install(
  ServiceModule<MovieService>()
    .dependsOn<DatabaseService>()   // doesn't mention the migration service at all
)

Startup order becomes: DatabaseService, then its enhancement SchemaMigrationService, then MovieService — and MovieService only had to declare a dependency on the database. Enhancements let infrastructure compose without leaking into every consumer’s dependency list. That separation is the kind of thing you only appreciate once a service graph gets big.

The ReadyService pattern

Here’s the clever bit, and the reason Misk’s shutdown is correct by construction rather than by hope.

Misk can’t know in advance which services your app uses, so it can’t hard-wire “Jetty depends on Redis.” Instead it divides the world into two kinds of service:

  • Work-generating services — things that ingest or create work: Jetty (HTTP), the SQS consumer, Cron.
  • Work-processing services — things needed to handle that work: JDBC, Redis, feature flags.

And it introduces a symbolic do-nothing service, the ReadyService, as the pivot. The rule, which you follow when wiring your own services:

Work-generating services depend on ReadyService. Work-processing services are enhanced by ReadyService.

From that single convention, the ordering falls out automatically. Startup:

  1. Work-processing services (Redis, JDBC) come up
  2. ReadyService flips
  3. Work-generating services (Jetty) come up

Shutdown walks the graph in reverse:

  1. Work-generating services (Jetty) stop first — no new work is accepted
  2. ReadyService stops
  3. Work-processing services (Redis, JDBC) stop last

That ordering is the whole game: by the time your database connection closes, Jetty has already stopped accepting requests and the requests it had already accepted have drained through. You get correct graceful shutdown without any service hard-coding knowledge of any other. In your own modules, the takeaway is simple — infrastructure enhancedBy<ReadyService>(), traffic sources dependsOn<ReadyService>().

What MiskApplication does at startup

Reading the actual MiskApplication.start() makes the sequence concrete:

val serviceManager = injector.getInstance<ServiceManager>()
serviceManager.startAsync()
serviceManager.awaitHealthy()        // block until every service is RUNNING
// only now start the health service, last:
jettyHealthService.startAsync()
jettyHealthService.awaitRunning()

Note what’s outside the ServiceManager: the JettyHealthService. Misk deliberately manages it separately so it can start last and stop last — it’s what answers liveness/readiness probes, so the container must only advertise “I’m alive” after everything real is up, and must keep answering during shutdown until everything has drained. If a service fails to start, awaitHealthy() throws, Misk tears the partial start back down, and the process exits non-zero. Fail-loud, again.

Graceful shutdown

When the process receives SIGTERM, Misk runs a shutdown hook that does the orderly teardown:

serviceManager.stopAsync()
serviceManager.awaitStopped()        // drains in dependency-reverse order
// then stop Jetty and the health service, last
jettyService.stop()
jettyHealthService.stopAsync()
jettyHealthService.awaitTerminated()

Because of the ReadyService ordering, awaitStopped() tears services down newest-dependent-first: traffic sources die, in-flight work finishes against still-running processors, then processors close. The health service is stopped dead last so orchestrators keep seeing a live container until the drain completes — stop advertising liveness too early and Kubernetes may SIGKILL you mid-drain.

The practical contract for you: when you write a Service, make its shutDown() actually finish or hand off in-flight work, and trust the ordering to keep your dependencies alive while you do. Don’t spin up your own shutdown hooks competing with Misk’s.

Warmup: readiness without the cold-start cliff

A freshly started JVM is slow: cold caches, empty connection pools, unallocated thread pools, and a JIT compiler that hasn’t found its hot spots yet. If you start taking production traffic the instant you’re “up,” those first requests eat the cold-start tax. misk-warmup exists to run production-like work before the service advertises readiness — seeding caches, filling pools, warming hot paths — so the first real request lands on an already-warm process.

Wire your warmup tasks through the warmup module and they run as part of coming-ready, between “services started” and “accepting traffic.” For a latency-sensitive service behind a load balancer that pulls from the pool the moment readiness flips, this is the difference between a clean deploy and a latency spike on every rollout.

Production notes & gotchas

  • Use enhancedBy for implementation details, dependsOn for real requirements. Migrations enhance a datasource; an app service depends on it. Modeling an enhancement as a dependency leaks infrastructure into every consumer.
  • Follow the ReadyService convention or lose graceful shutdown. If you bind a custom work-processing service without enhancedBy<ReadyService>(), it may stop before Jetty drains, and in-flight requests will hit a dead dependency. This is the most common self-inflicted shutdown bug.
  • Don’t report ready before you’re warm. Tie readiness to warmup completion for latency-sensitive services; otherwise every deploy is a small outage for the first cohort of requests.
  • Let Misk own the shutdown hook. It already installs one that drains in the right order and keeps the health service alive. Adding your own Runtime.addShutdownHook racing it is how you get half-drained shutdowns.
  • A failed start is a clean exit, not a zombie. If awaitHealthy() throws, Misk stops what it started and exits. Don’t catch and swallow startup failures hoping to limp along — a Misk service that can’t satisfy its dependency graph should die and let the orchestrator retry.

What’s next

Your service starts, reports ready, and dies cleanly. Now it needs something to do. In Part 5: Actions — HTTP Endpoints in Misk we’ll write real endpoints with WebAction, @Get/@Post, path and query params, and typed request/response bodies — and see how Misk’s action model compares to Spring’s @RestController.


Target keywords: misk service lifecycle, misk graceful shutdown, misk ReadyService.

Comments