Runbook: Redis Failover

The rate-limit Redis holds all counter state: windows, daily budgets, quota usage. It is the only store in the hot path. This runbook covers what happens when it goes away and how to bring it back.

Symptoms

• brume_ratelimit_degraded_total rising on the gateway's Prometheus metrics.
• Evaluations returning degraded: true with X-Brume-RateLimit-Degraded: true.
• brume_ratelimit_eval_duration_seconds elevated just before degradation (connection timeouts), then fast again (fail-open short-circuit).

Behavior while Redis is down

The gateway fails open: every evaluation returns success: true, degraded: true. Traffic keeps moving; checks are unaccounted. This is deliberate — availability over correctness — and it is visible in the response and the metrics rather than silent.

Customer-facing impact during the window:

• No denials from rate rules (everything is allowed).
• remaining/reset values in responses are not meaningful.
• SDK clients with timeout.fallback: 'deny' or onError: 'deny' will deny instead — that is their configured choice, not a gateway fault.

Recovery procedure

1. Confirm the gateway itself is healthy: /readyz, brume_api_requests_total steady.
2. Restore Redis (restart the service, fail over to the replica, or replace the instance).
3. Verify connectivity from the gateway host.
4. Watch brume_ratelimit_degraded_total stop incrementing and brume_ratelimit_eval_duration_seconds return to baseline.
5. Smoke test one limit and one check call against a known namespace.

After recovery

• Counters restart from zero. Window state lived in the lost Redis; the first window after recovery is effectively fresh. This is acceptable — the alternative (failing closed) would have dropped traffic entirely.
• Check brume_ratelimit_local_mismatch_total for any divergence between the in-process cache and Redis state; it should settle quickly.
• If the outage was long, daily budgets and quotas also reset with the Redis state. Note it in the incident record.

What never happens

• The gateway never blocks traffic waiting for Redis — it fails open instead.
• Postgres is unaffected; rules, keys, and configuration remain intact.
• No customer data is lost, because counters are ephemeral by design — see Security.