Health check design: how do I separate Liveness and Readiness?

Short answer: a static {status:ok} only proves the HTTP server is up, not that the app can actually serve. The fix is to separate two different questions: “is it alive?” and “is it ready?”.

Short answer

The real issue is this: Liveness and Readiness measure different things, and if you mix them up you shoot yourself in the foot — especially if you put the dependency check on the wrong probe. Readiness’ “pull me out of rotation but don’t kill me” behaviour is the same traffic-draining logic you need when a pod shuts down during autoscaling; I covered that side in the SIGTERM graceful shutdown answer.

Why

1. The two probes map to different consequences. If liveness fails, Kubernetes restarts the pod; if readiness fails, it just stops traffic. This distinction is critical: on a temporary dependency issue you don’t want to restart the pod, you only want to stop traffic.
2. Putting a dependency on liveness creates a restart loop. A momentary DB blip throws all your healthy pods into an endless restart loop — even though nothing is wrong with the processes themselves.
3. Probe cost multiplies by replica count. If every replica’s probe runs a real query against Postgres every few seconds, 50 pods × constant probes = you take down your own database.

What to do

1. Keep liveness cheap and dependency-free. Don’t check the DB, cache or queue here. Liveness should only ask “is the process responding?”.
2. Check the critical dependencies in readiness. Check the dependencies you need to serve here, like the DB, cache and queue. When a dependency goes down, let the Load Balancer pull the pod out of rotation — but without killing it. When the dependency comes back, the pod re-enters traffic.
3. Run readiness with a short timeout and cache its result. Cache the result for a few seconds, and check only what you truly need to serve.
4. Add a startup probe for slow boots. If your app boots slowly, use a separate startup probe so liveness doesn’t kill the pod while the app is still coming up.

Bottom line: I’d make liveness cheap and dependency-free, and readiness dependency-aware but cached. The golden rule: never let your probes DDoS the database. I separately discuss whether you really need Kubernetes on sade.dev; but if you don’t make this distinction, the smallest DB tremor will shake your whole cluster.