Runbook: Deploy Rollback

Deploys are zero-downtime: the new binary starts under systemd, passes readiness, and the old process is terminated. Because every check is a short HTTP request with no long-lived connections, a deploy never holds traffic hostage. This runbook covers the deploy itself and the rollback path.

Signals to watch during a deploy

All on the gateway's Prometheus endpoint (GET /metrics, token-protected):

• brume_ratelimit_eval_duration_seconds — p99 should stay within ~2x the pre-deploy baseline.
• brume_auth_failures_total / brume_auth_failure_rate_per_min — a spike means keys are being rejected; suspect an auth-cache or migration issue.
• brume_ratelimit_degraded_total — any rise means Redis connectivity regressed.
• brume_plan_limit_rejections_total — should track traffic, not the deploy.
• GET /readyz — must return ready before the old process is terminated.

Deploy procedure

1. Build the release binary and stage it alongside the running one.
2. Run migrations against Postgres (migrations are additive; the old binary must keep working against the new schema).
3. Start the new process under systemd.
4. Wait for GET /readyz to report ready.
5. Smoke test: run one POST /v1/ratelimit/check against a known namespace and confirm a sane verdict.
6. Terminate the old process.
7. Watch the signals above for 10 minutes.

Rollback

Rollback is a hard cut back to the previous binary:

1. Stop the new process.
2. Start the previous binary (kept staged from the last good deploy).
3. Confirm /readyz and re-run the smoke check.

Notes:

• Migrations are not rolled back. Schema changes are additive precisely so the old binary runs against the new schema.
• In-flight requests on the terminated process fail fast; clients should retry once. The SDK's onError/timeout fallbacks cover this.
• If the regression is a config change rather than code, revert the config and restart — same procedure, faster.

When NOT to roll back

• A rise in brume_ratelimit_denied_total alone is not a deploy regression — it usually means a customer's traffic changed. Check brume_ratelimit_eval_duration_seconds and error rates first.
• A single degraded blip during the cutover second is expected; sustained degradation is not.