Writing

Design the Degraded State Before the Happy Path

Design the Degraded State Before the Happy Path - abstract illustration

Most systems I've inherited have exactly two states baked into the code: working, and crashed. Anything in between gets forced into one of those two boxes, usually by throwing, which turns a partial problem into a full outage. The fix isn't a better exception handler. It's adding a third state to the model before you write the code that needs it, not after the first 2am page teaches you the hard way.

The two-state trap

A health check that returns true or false is answering the wrong question. Say your app has five components, an email sync, a scheduler, an image-generation client, a database. Say the image-generation credential is missing. The honest state of the system is "four things are fine, one specific thing can't do its job for a specific reason." A binary health check has no room for that sentence. It either reports healthy, which is a lie, or unhealthy, which takes down the whole app for a problem that affects one feature.

Throwing has the same shape. A missing credential deep in a startup path, treated as an exception, will either crash the app on launch, over one optional feature, or get caught somewhere and silently swallowed, which is worse: now nothing works and nothing tells you why. Neither outcome is what actually happened. What actually happened is narrower and more specific than either.

A third state: running, degraded, disabled

The fix is cheap once you see it: model three states instead of two. running means the component is doing its job. disabled means it isn't configured, and that's fine, plenty of features are opt-in. degraded is the state that matters most, and it carries a reason: not just "broken," but "SMTP not configured" or "missing IMAP credential" or "claude CLI not found on PATH." I built exactly this recently in an automation engine I'm rewriting: a registry of components, each independently running, degraded(reason), or disabled, surfaced on a status dashboard where every reason is a sentence, not a red dot.

The behavioral rule that makes this work is as important as the model itself: a missing credential or a misconfigured optional feature marks its own component degraded and leaves everything else alone. It never takes down the process. The scheduler still runs, the filesystem tools still work, and the one thing that can't do its job says so, specifically, in a place you'll actually look. I wrote about the app this pattern lives in, and the reasoning behind rewriting production infrastructure at all, in rewriting my own infrastructure as a product; this is one of the pieces of that build that generalizes past the one app.

Where the reason has to live

A degraded state without a reason is just a yellow light instead of a red one, marginally more honest and equally useless. The reason has to be specific enough that you, or someone who has never seen the code, can act on it without opening a debugger. "Scheduler degraded" tells you nothing. "Scheduler degraded: a configured job needs the claude CLI and it isn't on PATH" tells you exactly what to fix and where to look.

That specificity has to be decided at the point of failure, not reconstructed later from a stack trace. When the component that owns a capability discovers it can't fulfill its contract, that's the one moment it has full information: which credential, which config value, which external binary. Every layer above that either drops the detail or has to guess. So the reason gets attached right there, as a string that's part of the state, not logged and hoped for.

This is a design decision, not a bug fix

The reason this has to be decided early, before the happy path, not patched in after: retrofitting a third state into code that only knows two is close to a rewrite of every call site that currently throws or returns a boolean. Every place that decides "is this thing okay" has to change its answer type. Building the registry first means every component, from the first one you write, reports into a shape that already has room for "yes, but," instead of forcing that answer through a boolean gate that was never built to carry it.

It also changes what "done" means for a feature. A component isn't finished when the happy path works. It's finished when you can answer, for every way it can fail to be configured or reachable, what state that produces and what reason gets attached. That's a small amount of extra design time per component, decided once, instead of an unbounded amount of debugging time later, spent every time someone hits a failure the two-state model couldn't describe.

The test that proves it

The way to know this is actually working, not just modeled, is to break something on purpose and watch what happens. Pull a credential, misconfigure a path, remove a binary from PATH, then launch the app. The bar is simple: everything unrelated keeps running, the dashboard names the one thing that's degraded and says why in a sentence a non-engineer could read, and nothing crashed. If any of those three isn't true, the model isn't actually load-bearing yet, it's decoration.

A two-state system optimizes for the case where nothing is wrong. A three-state one optimizes for the far more common case, where one specific thing is wrong and the other nineteen are fine. Most of what breaks in production is the second kind. Design for it before the first component ships, and the difference shows up not as an outage, but as a sentence on a dashboard that tells you exactly what to go fix.

Building something like this?

This is the kind of work I do for clients. Tell me what you're building and I'll give you a straight read on the approach.

Book a 30-minute call