Stop Grading AI on Vibes: Lightweight Evals for Real Work

Here's how most people check whether an AI change worked. They run the prompt once, read the output, and decide it looks about right. Then they ship it. I did this for longer than I'd like to admit, and it works fine right up until it doesn't, because "looks about right" is a feeling, and a feeling can't tell you that today's output is 5% worse than last week's. You can't feel a small regression. You can only measure one.

The fix isn't a heavyweight testing framework or a research lab's eval suite. It's the same instinct that makes you write a test after you fix a bug: capture what "good" looks like on a handful of real cases, and re-run those cases every time you change something that could move the behavior. That's a lightweight eval, and for a fleet of agents where a silent behavior change costs real money, it's the difference between catching drift in a two-minute run and finding out from a downstream mess a week later.

Why vibes-based grading fails

Reading the output and nodding feels like verification. It isn't, and it fails in three specific ways.

You can't perceive a small regression. If a prompt used to get 19 of 20 cases right and now gets 17, the two it newly breaks won't be the ones you happen to eyeball. You'll read one output, it'll look fine, and you'll conclude nothing changed. A 10% drop in quality is invisible to a spot check but obvious to a scored run.

You can't compare two prompts fairly by memory. When you're tuning a prompt, you try version A, read it, try version B, read it, and then decide B "feels" better. But you're comparing B's fresh output against a fading memory of A's, on whatever single input you happened to use each time. That's not a comparison. Run both against the same fixed set of inputs and the winner stops being a matter of recall.

You forget what good looked like. Last week's output that impressed you is gone. You have no artifact of it, so when this week's output is slightly worse you have nothing to hold it against. Evals give you that artifact: a frozen record of the expected result you can diff against forever.

None of this means human judgment is worthless. It means human judgment is the thing you use to build the eval once, not the thing you rerun by hand every time you touch a prompt.

What a lightweight eval actually is

Strip away the tooling and an eval is two things: a set of representative inputs, and a way to check the output against a known-good expectation. That's it.

The inputs are a dozen or so cases that look like your real workload: a few typical ones, a few edge cases, and every case that's ever burned you. The checker is whatever's cheapest and honest enough for that case. In practice it's one of three:

Exact or structural match. The output has to equal a known string, or parse as valid JSON with the right fields and types. This is the gold standard when it applies, because there's no argument about whether it passed. If your agent is supposed to return structured data, you can assert on the shape directly.
A rubric check. For open-ended output where no single string is "the" answer, you write down what a good answer must contain and check against that. "Mentions the refund window. Doesn't invent a policy. Stays under 120 words." A rubric turns a vague sense of quality into a checklist you can actually score.
A second model as judge. For genuinely subjective output, you hand the input, the output, and the rubric to a separate model and ask it to score the result. This is the same move as having AI review AI: the model that judges isn't the model that produced the work, and the judge runs against a fixed rubric so its scoring is consistent across runs. It's not perfect, but it's repeatable, which is the whole point.

You don't need all three. You need the cheapest one that gives you a signal you'd actually trust. Most of my useful evals are exact-match or structural, because I've deliberately pushed my agents toward returning data instead of prose, and data is checkable.

Build the set from failures you've already hit

The best eval cases aren't hypothetical. They're the bugs you've already lived through. Every time an agent does something wrong in production, that specific input is a test case waiting to be captured, and the correct output is the fix you just made.

This is the exact same instinct as keeping a gotchas file your AI reads before it repeats your mistakes. The gotchas file stops the model from re-making a known error; the eval proves it didn't. They're two sides of one discipline: when something breaks, you don't just fix it and move on, you make the fix permanent by recording the case.

Concretely, when my CRM email sync miscategorized a certain kind of forwarded message, I didn't just adjust the prompt. I saved that message as an eval input with the category it should have produced. Now that case runs on every change to that agent, and if a future prompt tweak or model upgrade re-breaks it, I know before it ships instead of after. A regression suite built this way is self-assembling: you never sit down to write a hundred test cases, you accrue them one real failure at a time, and every one is guaranteed relevant because it actually happened.

Seed the set with a few obvious happy-path cases so a total break is caught too, then let production failures do the rest of the work.

Run them when something changes

An eval set only earns its keep if you run it at the moments behavior can move. There are three, and they map to the boundaries where I already treat AI work like code.

When you change a prompt. A prompt edit is a code change to your most important logic, so it gets the same treatment: run the eval before and after, and read the delta. If you improved case 4 but quietly broke case 11, the score tells you. Without the eval, you'd have shipped the trade blind.

When you swap a model. A newer, more capable model is not a drop-in. It can be better on average and still regress the one load-bearing case your prompt was implicitly tuned against. The eval set is exactly what you run the candidate through before you move anything into production.

When you upgrade a pinned version. This is why I pin my model versions in the first place. Because the version is pinned and explicit, a model change is a deliberate act with a diff, and the eval run is the gate that act has to pass. New version ships, I run it against the real prompts and the eval set in a non-production config, I read the score, and I only move the pin if it holds. The eval is what turns "upgrade on purpose" from a slogan into a checkable step.

The connective tissue is that all three are changes you'd never make to real code without a test run. The model, the prompt, and the version are dependencies. Evals are how you catch drift at the boundary instead of discovering it downstream.

Keep it cheap and keep it honest

The failure mode of evals is over-building them into a project of their own. Don't. A dozen good cases that reflect real failures beat a hundred lazy ones that all test the same easy path. Coverage that looks impressive but exercises one code path is worse than a small set that hits your actual sore spots, because the big set gives you false confidence.

Honesty matters as much as size. When you skip a case, log that you skipped it and why, rather than deleting it and pretending your suite is green. A muted case with a note ("flaky judge, revisit") is information; a silently removed one is a lie you'll tell yourself later when you think coverage is complete. The same goes for the judge model: spot-check its scoring occasionally, because a rubric-based judge that's quietly too lenient will hand you a passing suite while quality slides.