Tag
Staying up under failure: timeouts, retries, backoff and bulkheads that contain the blast. Questions under this tag, answered directly.
14 answered questions carry this tag.
Hold the new version at a canary weight and check 5xx and latency against a threshold; on breach, roll back to the last stable release and alert the team.
Make Canary the default for routine releases and save Blue-Green for big cutovers needing an instant flip, since both demand expand/contract migrations.
Take a periodic base backup, archive WAL to S3 continuously, replay to a recovery_target_time seconds before the bad statement, and rehearse the restore.
Put `tries` and a `backoff` on the job so it lands in `failed_jobs` at the cap, alert from a `JobFailed` listener and replay it with `queue:retry`.
Keep liveness dependency-free and check dependencies in readiness with a cached probe, since liveness restarts the pod while readiness only sheds traffic.
Acquire locks in one global order such as ascending PK, keep transactions short, target `FOR UPDATE` at the fewest rows, and retry the rest with backoff.
Wrap requests in a `defer recover()` middleware at the outermost layer, log the stack with the request id, return a generic 500 and emit a metric.
RPO 5min buys continuous cross-region replication and PITR, RTO 30min an IaC-provisioned warm standby behind Route 53 failover, rehearsed on a game-day.
Collapse repeated errors into one counted line, back off retries, cap logrotate by size and count, and give `/var/log` a volume off the root disk.
Per-IP limits lose to rotating proxies, so key the Redis sliding-window on account + IP + ASN, add account lockout and leave volumetric floods to the edge.