I added a new permission domain and went to update the tests that restrict what a limited key can hold. Updating them meant editing a hardcoded list of forbidden domains inside the test file itself. That edit is the tell: if adding a value to the production constant requires me to also add it to the test by hand, the test isn't reading the production constant at all.
What the test appeared to guarantee
The setup is a simplified permission mode that issues API keys from a short list of checkboxes. Some permission domains must never appear on a key issued that way, ever, and there's a constant naming them.
The tests around it were cartesian assertions: build every combination of checkbox selections, generate the resulting grants, assert that no forbidden domain shows up. That reads like exactly the right test. It's the one I'd point at if someone asked what stops a restricted key from holding a dangerous permission.
Both assertions declared their own local list of forbidden domains, because the real constant was file-private and not visible to the test target. So the test compared the output against a copy of the answer rather than against the answer.
Why the copy makes it worthless
Remove a domain from the real forbidden list and the test still passes. It passes because nothing in the generation path emits that domain anyway: no checkbox maps to it, and the grant builder wouldn't produce it either way. The assertion never had to consult the real list to be satisfied.
So the test proves that grants generated from checkboxes don't include things no checkbox generates. Which is true, and circular, and not what anyone reading the test name would think it covers.
That's the part I want to underline. A test that fails to catch a regression is a gap. A test that reads as coverage of a specific safety property, is named for that property, and does not actually check it is worse than nothing, because it stops anyone from writing the real one. I'd rather have an obvious hole than a decoy.
The fix isn't clever: expose the real constant to the test target so the assertion consults it, and the test fails the moment someone removes a domain from the list. There was already a small test elsewhere doing exactly that, added almost incidentally. That one was the shape to extend. The impressive-looking cartesian assertions were the ones to distrust.
The second one: a fixture with its own bug
The same week, a set of tests around filesystem permissions was failing and I went looking for the production bug. There wasn't one. The fixture was constructing grants with the wrong identity kind, naming resources by string where the real system identifies them by id.
That's the more insidious version, because the fixture bug and the production behavior can cancel out. A fixture that builds permissions slightly wrong can make a passing test pass for the wrong reason for months, and you only find out when something adjacent changes and the arrangement stops lining up.
Both cases share a root: the test was written against a mental model of the system rather than against the system. The forbidden-domain list existed in the author's head and got typed into the test. The grant identity model existed in the author's head and got typed into the fixture. Neither one was read from the code.
The tell is the manual edit
There's a cheap heuristic in both stories. When you change a production value and a test needs a matching hand edit to keep passing, stop and look at why.
Sometimes the answer is fine: the test is pinning an exact expected output and that's the whole point. Often the answer is that the test contains a mirror of the thing it's supposed to be checking, and the mirror will drift. The moment you're maintaining two copies of a value, one of them is going to be wrong eventually, and it'll be the copy that nothing depends on at runtime.
The same tell shows up in a slightly different form with docs. I had a doc comment claiming a function was a byte-range splice when it had never been one, and a scheduler doc naming a method that didn't exist. Nothing enforces a comment. But a test is supposed to be enforcement, which is why a test that's really just a comment is a category error.
Write the failing case first, even for a guard
The way I'd have caught both of these at the time is a discipline I'm inconsistent about: before trusting a test that asserts something is prevented, break the prevention and watch it fail.
Delete the domain from the forbidden list and run the suite. If it stays green, the test isn't testing that. That takes thirty seconds and it's the only reliable way to know a safety test is wired to the thing it claims to guard.
It's the same instinct behind having AI review AI, and behind checking that a parser's fixtures resemble real input. A test suite tells you your assumptions are self-consistent. Whether they match the system is a separate question and it needs a separate check.
What I changed in how I file these
Both of these went into my punch list as items with a named regression test attached, because an item without one isn't ready to close. The forbidden-domain one is written as a requirement rather than a description: removing a domain from the list must fail a test. That's a statement someone can verify in a minute.
Neither was severe. One was a minor test-quality issue, the other a fixture bug that surfaced as noise. But the reason I bothered writing them up carefully is that the cost isn't in the bug, it's in the confidence. The test I'd have pointed at to answer "is this safe" was the one that couldn't answer it, and I'd have pointed at it without hesitating.
The short version
A test that hardcodes a copy of the value it's checking is documentation with a green checkmark. The value is that a green suite means something; a decoy test spends that value without providing it.
If you have safety-critical constants, make the tests read them. Then break one on purpose and confirm the suite notices. Anything you can't break and watch fail, you don't actually know is covered.