Reverse proxy restart blast radius calculator

Where the defaults come from

Every default above was read off one production host on 9 August 2026, during an outage that had been running for days without anyone noticing it as an outage. Ten hostnames were served by a single reverse proxy process with three backend replicas behind it. The proxy was restarting roughly every two to nine minutes.

The reason it took a while to see is that each restart lasted about a second. Nothing was down long enough to trip a check that polls every thirty seconds, and recovery was automatic, so what showed up was a thin stream of unexplained 502s that looked like flaky clients.

Four consecutive recoveries, measured from the crash line to the line that says the proxy is accepting requests again:

At sixteen restarts an hour, 1.15 seconds each, the arithmetic in the calculator gives 18.4 seconds an hour, 7.4 minutes a day, and an availability ceiling of 99.489%. That is not "about three nines". Against a 99.9% target it spends about five times the entire monthly error budget, and it does so before a single backend failure is counted.

The part that was actually wrong

The first three diagnoses were all wrong, and each was wrong in a way worth naming, because they are the obvious guesses.

It was not a segfault. The restart count was in the fifteen hundreds, which looks like a crash loop, so a native fault was assumed. But dmesg had no segfault record for the process, and the container's exit code was 0.

It was not an out-of-memory kill of the proxy. The proxy's own cgroup reported oom_kill 0 and was using 566 MiB of a 4096 MiB limit. It had enormous headroom. Reading memory.events inside the right cgroup settles this in one command, and it is worth doing before touching a limit.

It was not fixed by upgrading. Five newer versions existed. Extracting the binary from the newest one and running strings over it showed the same vulnerable dependency version as the running build, so an upgrade would have changed the version number and nothing else.

What it actually was: the proxy resolved its upstreams by hostname, and the library it is built on resolved that hostname on the request path with an unwrapped call. In Rust an unwrapped error in a worker thread aborts the process. The relevant line, in the dependency's own source, carries a TODO next to the unwrap noting the error is unhandled. So a momentary failure of the container DNS resolver did not produce one failed request; it produced a dead proxy and ten dead sites.

The trigger was memory pressure elsewhere. A backend was being killed at its memory ceiling around fifty-six times in three hours, and while it restarted its name briefly stopped resolving. Fourteen out of fourteen proxy restarts in the sampled window followed a backend kill by one to two seconds. A database in a different cgroup was also being killed in the same window and never triggered a proxy restart, which is the control that makes the link specific rather than "the host was busy".

Why three replicas bought nothing

This is the part the calculator exists to make concrete. Three replicas is a real answer to "what if a backend dies", and the backends were dying constantly, and the replicas handled it exactly as intended. It made no difference. The ten sites did not share three of anything that mattered; they shared one process, and that process exited.

Redundancy below a single point of failure only moves the failure. It is worth checking, for anything you run, whether the layer you have made redundant is the layer that is actually failing.

The fix, and its limits

Passing IP literals rather than hostnames removes the lookup from the request path, which makes the crash unreachable rather than less likely: an address that is already an address cannot fail to resolve. Routes pointing at services that no longer existed were sent to an address that refuses connections, so those return 502 for that route instead of killing the process for every route. Because container IPs were not pinned, a small reconciler keeps the rendered configuration in step with reality and the hostnames remain the source of truth.

Verification was a deliberate test rather than a quiet period. Restarting a backend on purpose reproduces the exact condition that used to kill the proxy; afterwards the proxy's start time was unchanged and every site stayed up. Waiting and seeing nothing happen would not have shown this, because the trigger had stopped firing on its own during the observation window — which is the trap in declaring an intermittent fault fixed.

What this did not fix: the backend still exceeds its memory ceiling. That host carries container limits summing to roughly 2.2 times its physical RAM with swap 87–94% full, and the kill rate was climbing through the day from nine an hour to twenty-five. The proxy now survives it, so the ten sites stay up, but the capacity problem underneath is untouched and no configuration change fixes it. Separating "stop the outage" from "fix the cause" is usually the right order, as long as you are honest that you have only done the first.