Benchmarks

This page describes how we measure Brume, so you can judge any number we publish — and reproduce the measurement yourself. We do not publish numbers we cannot stand behind. The harness ships with the repository (brume-bench); the first measured results are below.

What a "check" measures

The unit under test is one POST /v1/ratelimit/limit evaluation: auth lookup, rule resolution from the in-process cache, daily-budget check, and the atomic Lua evaluation against Redis — the full path your application exercises.

We measure:

• Evaluation latency (p50 / p90 / p99 / p999), client-observed. These figures are taken over loopback HTTP (plaintext); TLS termination is not included.
• Sustained throughput (checks/second) at a fixed error budget.
• Degraded behavior: latency and correctness when Redis is removed from the path (fail-open). Measured below — every request allowed with degraded: true, zero denials.

Workloads

First measured results

Measured 2026-08-29 on the production artifact — plain release build cfb6b48 — socket-activated on 127.0.0.1:8422 against the bench database, 30-second runs after a 5-second warmup, zero errors (no HTTP failures, no decode failures, no degraded responses) across all four workloads. Hardware: see below.

Throughput rows (documented concurrency)

Notes on reading these:

• bursty-login is denial-heavy by design. At ~5.2K checks/s against 1,000 IPs, every IP blows through its 5-per-minute window within the first second, so the measured phase is a pure deny-path exercise (the blocklist is in the rule set and was behaviorally verified). Per-identifier isolation was verified separately: two fresh IPs each start with a full bucket.
• hot-key at concurrency 500 is host saturation, not service latency. The 4-vCPU host is shared by client and service; 500 in-flight workers on 4 cores means queueing dominates — that is what a saturated single machine looks like, and it is the honest number for that shape. The low-concurrency rows below show the service latency underneath.
• multi-rule counts calls, not evaluations. Each call evaluates two rules (per-minute + per-day) dispatched together on one Redis connection, sharing one round trip: 3,960 calls/s ≈ 7,920 evaluations/s at concurrency 100.

Service-latency rows (concurrency 4, 15-second runs)

Single-tenant-shaped load: four in-flight requests, same host, loopback.

Cold start (steady-api, no warmup, first traffic after gateway restart)

Cold start costs ~0.2 ms at p50: auth and rule caches load per-project on first use, not per-request. The gateway was restarted and hit immediately with 15 seconds of traffic at concurrency 4, no warmup.

Degraded mode / fail-open (rate-limit Redis unreachable)

The gateway points at an unreachable Redis endpoint; every check takes the fail-open path.

Fail-open is faster than the enforcing path (0.3 ms vs 2.3 ms at p50, c=4) because the Redis round trip is the dominant cost; when Redis is unreachable the check resolves from the auth and rule caches alone. Every response carried degraded: true and allowed the request — zero denials, zero errors. The tradeoff is stated plainly: during a Redis outage Brume stops enforcing limits and keeps serving; enforcement resumes automatically on reconnect.

Hardware

The service runs on a single VPS in Frankfurt (see Deployment). These figures were measured on that host itself, over loopback, with the bench client sharing the machine with the service:

• 4 vCPU (QEMU guest), 11 GB RAM, Linux 6.8
• Postgres remote; Redis in-region; the Lua evaluation round-trip is inside every measured latency
• Client: brume-bench, 4 worker tasks maximum in the service-latency rows, HDR histogram per worker

We publish the machine class with every result — a number without its hardware is marketing, not measurement. Same-host loopback is disclosed here because it is what was measured; numbers from a separate client machine in the same region will replace these when measured, with both the client and server cost visible.

Rules we hold ourselves to

1. No cherry-picked percentiles. p50, p99, and p999 ship together.
2. Warm and cold. Rule-cache warm and post-restart cold starts are reported separately — both rows are above.
3. Fail-open is measured, not assumed. Degraded-mode latency is its own row — above.
4. Reproducible. The harness, the config, and the exact build are published with each result.

Reproducing

The harness lives in the repository at crates/core/src/bin/bench.rs and builds as brume-bench:

# 1. Provision the bench project, API key, rules, and blocklist in Postgres.
#    Run BEFORE starting the gateway: the rule cache has no TTL, so rules
#    inserted while the gateway is up are invisible until restart.
brume-bench seed --db-url postgres://...

# 2. Start the gateway against the bench database, then drive a workload.
brume-bench run --url http://127.0.0.1:8422 --key <key> \
    --workload steady-api --concurrency 100 --duration 30 --warmup 5

Every result above was produced by exactly those two commands with the flags shown in the table rows. New numbers land here and in the changelog — with the machine, the workload, and the build.