How do I do graceful shutdown on SIGTERM during autoscaling scale-in?

Short answer: scale-in cuts work off mid-flight because the app doesn’t drain on SIGTERM — what you need to fix is the process’s shutdown lifecycle.

The real issue isn’t autoscaling, it’s the shutdown protocol: when the platform sends SIGTERM, your app either dies instantly or gets cut off before finishing. The right order is: stop the traffic first, finish in-flight work next, die last.

1. Catch SIGTERM and stop accepting new requests. The first move is to fail your readiness probe or deregister yourself from the load balancer so it stops routing new requests to you. Don’t close existing connections yet — just stop the intake.
2. In Go, use server.Shutdown(ctx). Catch SIGTERM with signal.NotifyContext and hand http.Server a context with a deadline: server.Shutdown(ctx). It refuses new connections and lets open requests finish until the timeout. End ListenAndServe this way, not with kill -9.
3. In Octane, use a graceful stop/reload. Bring the Octane process down with octane:stop/graceful reload; workers finish the current request and stop claiming new ones. On the queue side, queue:work handles SIGTERM itself as long as you don’t kill -9 it — it finishes the running job and stops claiming new ones.
4. Align the platform’s grace period with your drain. Use an ASG lifecycle hook or k8s terminationGracePeriodSeconds to set how long the platform waits before killing to your longest acceptable drain time. Otherwise the platform fires SIGKILL after SIGTERM and chops the drain in half.
5. Make long jobs idempotent/resumable. So that even if the grace period runs out and the process is hard-killed you’re still safe, design long jobs to be re-runnable — every step idempotent, so an interrupted job can safely restart or resume.

Bottom line: order matters — drain the LB first, then stop new intake, then finish in-flight work, then die. In Go that’s server.Shutdown(ctx); in Octane it’s a graceful reload plus queue:work’s native SIGTERM behavior; on the platform side, align the grace period with your real drain time. Make jobs idempotent on top of that and even a hard kill won’t corrupt data. I covered why Octane’s persistent-process model changes this shutdown behavior in more depth in the hub post.