A Test That Cannot Fail: Checking Whether Your Green Suite Checks Anything

We have ten regression tests for the name generator. All ten were green.

That is not evidence. A test suite reports two things — that the code passes, and that the suite is capable of noticing if it did not — and running the suite only ever tells you the first. The second has to be established some other way.

So we established it the only way that actually works: we broke the code on purpose, one thing at a time, and checked that the right test went red. Eleven deliberate defects. Ten were caught. The eleventh could not be caught, for a structural reason worth more than the ten successes combined.

And along the way we found that one of our tests had, at an earlier point, been green on broken code — not through carelessness in the code under test, but through a two-character mistake in the test's own fixture.

The method

The mechanics are unglamorous and take about an hour to build. For each defect:

1. Copy the whole source tree to a scratch directory.
2. Apply exactly one mutation to the copy — one file, one edit.
3. Run the suite in the copy, restricted to the test that is supposed to catch it.
4. Assert the status is FAIL.
5. Throw the copy away.

Then, as a control, run the suite once on an unmutated copy and confirm it is entirely green. Without that control you cannot distinguish "the test caught the mutation" from "the test was broken in the scratch tree all along."

Note what step 3 asserts. Not "some test failed" — that specific test failed. A mutation that corrupts a data file will often trip three tests at once, and if you only check for a non-zero exit code you will credit the wrong test with the catch and keep believing in a test that is actually dead.

The eleven mutations

Each one is a plausible defect, not a random character flip. That distinction matters: mutating < into <= in a loop bound produces defects nobody would ever write. What you want is the shape of mistake that has actually reached production somewhere.

Mutations 1 and 5 are the two defects most worth dwelling on, because both are real. Mutation 1 is a synthetic version of a defect we actually shipped into staging: a locality file whose weights were metropolitan-area figures rather than city figures, which made sixty Taiwanese cities sum to 106.9% of Taiwan. Mutation 5 is what a broken cohort build looks like — twenty-year-olds silently receiving the given names of eighty-year-olds, with every name in the output perfectly real.

The result: ten of eleven mutations produced the expected FAIL. One did not, and it was mutation 8.

The test that was green on broken code

Test 1 checks an invariant that everything else depends on. The picker must leave the mt_rand stream in the same position whether or not the dataset it was asked for exists. If it does not, then adding a single data file to the tree changes every field of every generated card, and every permalink the site has ever emitted resolves to different data.

The implementation of the invariant is a save-and-restore around the branch: draw one number, do whatever the branch requires, then mt_srand($keep) to put the stream back.

The first version of the test looked entirely reasonable. Seed the generator, call the picker with a dataset that exists, record the next mt_rand(). Seed it again, call the picker with a dataset that does not exist and a fallback, record the next mt_rand(). Assert the two match.

The fallback was written as:

fn() => 'X'

That fallback never touches mt_rand. And that is the whole defect — in the test, not the code.

With a fallback that consumes zero draws, both branches consume exactly one draw in total. The stream lands in the same place afterwards whether or not the reseed exists. The assertion is satisfied by arithmetic rather than by correctness. The test was green, and it would have stayed green with mt_srand($keep) deleted from the file entirely.

The real fallback is a faker call. Faker consumes an unknown and variable number of draws — that is precisely the condition the reseed exists to neutralise, and precisely the condition the fixture had eliminated. The fixture had accidentally simplified away the only thing that made the test meaningful.

The fix is to make the fallback deliberately hungry, and hungry by differing amounts:

$burn = fn(int $k) => function () use ($k) {
    for ($i = 0; $i < $k; $i++) mt_rand();
    return 'X';
};

mt_srand(4242); ng_pick_first(['t/weighted', 't/flat'], $burn(3));  // dataset present
mt_srand(4242); ng_pick_first(['t/NEMA1', 't/NEMA2'],   $burn(3));  // absent, fallback burns 3
mt_srand(4242); ng_pick_first(['t/NEMA1', 't/NEMA2'],   $burn(10)); // absent, fallback burns 10

Three calls, one assertion: the next mt_rand() must be identical in all three. The third call is the one that carries the weight. Present-versus-absent could pass by coincidence; absent-burning-3 versus absent-burning-10 cannot, because only a reseed can make two different appetites land in the same place. Mutations 9 and 10 both go red against this fixture. Against the old one they would not have.

The general lesson is not about random number streams. It is that a fixture designed for convenience will tend to erase the exact variation the test exists to detect, and it does so silently, because the result is a passing test.

The mutation that cannot be caught, and why we wrote that down

Mutation 8 deletes the same reseed from ng_pick(), and test 1 stays green. This is not a gap to be fixed later. It is a structural limit of the comparison the test performs.

ng_pick() has no fallback. There is nothing to call when the dataset is missing; it returns null. So both branches — dataset present, dataset absent — consume exactly one draw, and the present-versus-absent comparison is once again satisfied regardless of whether the reseed is there. The trick that rescued ng_pick_or() and ng_pick_first() is unavailable here, because there is no second appetite to vary. No amount of cleverness in the fixture changes this; the two branches are genuinely indistinguishable by stream position.

We recorded this in a comment directly above the test:

If you remove mt_srand($keep) from ng_pick() specifically, this test stays green, and that is a structural limit rather than an oversight. Do not treat a green test 1 as proof that the reseed in ng_pick() is present.

The reseed in ng_pick() is therefore held in place by code review and by a comment in the source — not by the suite. That is a worse guarantee, and saying so is the point. A blind spot you have written down is a known risk. The same blind spot undocumented is a test suite that lies to whoever reads its green output next, most likely you in eight months.

The other kind of test that cannot fail

Mutation checking finds tests that are dead by accident. It also draws attention to tests that were never alive, and we found one of those in our own suite.

We had a check comparing three quantities per locale: the share of a corpus held by its top ten entries, the corpus's claimed coverage of the national population, and the resulting top-ten share of the population. For Taiwan the suite currently prints 52.81% of the corpus, times 99.9% claimed coverage, giving 52.73%.

The reassuring part is the first number. Taiwan's Ministry of the Interior publishes a top-ten share of 52.79% for the full registry. Our corpus computes 52.81%. Two hundredths of a percentage point apart — exactly the kind of agreement that makes you stop looking.

The two-hundredths gap is itself explainable rather than mysterious, and worth pinning down because a vague "roughly 52.8%" is how a real discrepancy would hide. The registry's 52.79% is over 2,731 surnames and 23,373,283 bearers. Our 52.81% is over 2,707 surnames and 23,367,536 bearers, because we drop 24 unusable rows: the service category 其他 ("other", 5,174 bearers, not a surname) and 23 rows in the Private Use Area, where the ministry encodes rare characters that have no Unicode assignment and would render as blank boxes. That is 5,747 bearers, 0.0246% of the population, removed from the denominator. A slightly smaller denominator with the same top ten gives a slightly larger share. The numbers agree because the cleaning was small and correct.

None of which makes the identity check a test. It is worthless as verification, and worthless for a reason visible without running anything. Both sides are computed from the same stored weights. Claimed coverage is the sum of those weights over the national population; the top-ten share is the top ten of those same weights over their sum. The identity reduces to:

(a/b) × (b/c) = a/c

which is true for any a, b and c whatsoever. Fill the corpus with fabricated numbers and the check still agrees to two decimal places. It cannot go red on bad data, because it never looks at whether the data is good — only at whether division works.

There is one thing it can catch: a future refactor computing coverage by a different route, which would break the arithmetic identity itself. That is a real if narrow value. So the check stayed, but it was demoted out of the test set and into informational output, with a fixed PASS status and a warning printed above the numbers stating that this is self-consistency and not verification.

That demotion is the right move whenever you find one of these. Deleting the check loses a useful number. Leaving it as a test inflates the suite's apparent coverage with a row that can never fail. Printing it as information keeps the number in front of your eyes and keeps it out of the count of things that are actually being verified.

There is a postscript here that belongs in an article about tests that quietly stop meaning things. The explanatory comment above that check quoted a worked example: 48.14% of the corpus times 109.8% claimed coverage gives 52.84%. When we ran the check while writing this article, it printed 52.81%, 99.9% and 52.73% instead. The comment was not wrong when it was written — it described a build in which the Taiwanese corpus was truncated to its 401 most common surnames, so the weights overstated the population and coverage came out above 100%. The corpus now ships all 2,707 surnames with real bearer counts, and coverage is just under 100%. Nothing broke; the prose simply stopped matching the code and nothing in the world would ever have told us. Comments carrying numbers are unverified assertions that look like documentation, and they rot in exactly the direction where they remain plausible.

What a green suite is worth

Three states are easy to confuse and worth separating explicitly:

Rows three and four are indistinguishable from row one by any amount of running the suite. Only mutation distinguishes them, because mutation is what deliberately puts you in row four and asks what colour comes out.

This is also why coverage percentages are a poor proxy. Every one of our eleven mutations lives in code that was already covered — executed by the suite, counted in the percentage. Coverage measures whether a line ran. It says nothing about whether anything would have complained if that line had been wrong. Our old test 1 executed the reseed on every run and asserted nothing about it.

The checklist

1. Break the code and demand a specific failure. Not "the suite went red" — this test went red. Otherwise a broad mutation lets a dead test take credit for a live one's catch.
2. Run an unmutated control. A green control is what makes the red results mean anything.
3. Mutate one thing at a time. Two simultaneous mutations mask each other, and you learn nothing about either.
4. Choose defects someone would plausibly ship, not random operator flips. Wrong units in a data column, a stale cache, a shifted index, a wrong encoding.
5. Audit your fixtures for accidental simplification. A fallback that does nothing, a stub that returns a constant, a fixture with one row where production has thousands — these are how a test quietly stops being able to fail.
6. Vary the thing the code is supposed to be insensitive to. If the code claims robustness to how much the fallback consumes, the fixture must consume different amounts. One value tests nothing about invariance.
7. Look for identities masquerading as checks. If both sides derive from the same source, write out the algebra. If it reduces to something true for all inputs, it is not a test.
8. Demote rather than delete a check that cannot fail: keep the number as informational output, remove it from the count of tests.
9. Write down what your suite cannot see. A documented blind spot is a managed risk. An undocumented one is a false guarantee with your name on it.

We did not run this exercise because we suspected the suite. We ran it because a green result is unfalsifiable by construction, and the only honest response to an unfalsifiable result is to go and construct the falsification yourself. It cost an afternoon, it found one dead assertion and one identity pretending to be a test, and it turned a vague confidence into a specific, written-down claim: ten defects this suite catches, and one it does not.

For the invariant itself and why the generator saves and restores the stream, see How Our Generator Works. For the real defect that mutation 1 imitates — sixty cities summing to 106.9% of a country — see Sixty Cities, 106.9% of a Country. For the self-consistency trap in its original context, see Data Sources: Taiwan.

Data as of 2026-07-18

All figures were re-derived on 18 July 2026 from the files on disk, and the mutation run was executed on the same day against the shipped tree.

Sources and notes:

• Test suite — ng_regression_tests.php, shipped in source_share_1.1.8/set/. Ten tests, keyed 1–10 in a registry at the top of the file: (1) mt_rand stream invariant, (2) card determinism by seed, (3) weight distribution and flat-file uniformity, (4) empirical versus theoretical top-10, (5) sum of city populations against national population, (6) denominator identity (INFO, not verification), (7) city ↔ postal-code pairing, (8) close-tag-in-line-comment trap, (9) data encoding (BOM, UTF-16, CRLF, malformed UTF-8), (10) birth cohorts genuinely separating generations.
• Mutation harness — ng_regr_mutation_check.php, a working script rather than a shipped artefact. Eleven mutations as tabulated above; copies the tree with robocopy, applies one mutation, runs the suite with --only=N, and asserts the parsed status of test N is FAIL. Ends with an unmutated control run.
• The run of 18 July 2026 — ten caught, one missed. The miss is mutation 8 (ng_pick), where test 1 reported PASS because both branches showed the same next mt_rand value; it is a miss by construction, not by accident. Control run on an unmutated copy: tests: 10 · PASS 10 · FAIL 0 · WARN 0 · SKIP 0 · ERROR 0 · time 15.1 s.
• The fallback defect — recorded in the source at the fixture, which states that a fallback not touching mt_rand makes both branches perform exactly one draw, so the stream matches even with no reseed in the code, and that this is how the test was once green on broken code. The current fixture burns 3 and 10 draws respectively.
• The structural limit — recorded in a comment above test 1, and described there as verified by mutation rather than assumed. ng_pick() has no fallback branch, so present-versus-absent cannot distinguish the presence of the reseed.
• The denominator identity — test 6 carries a fixed PASS and prints a warning that it is self-consistency, not verification. Its live output on 18 July 2026 is zh_TW lastname_male · 52.81% top-10-in-corpus · 99.9% coverage · 52.73% product, re-run for this article. The 48.14% / 109.8% / 52.84% example still in the comment above that test describes a superseded 401-entry build and is stale; data-sources-taiwan.md was corrected on 18 July 2026 and now carries the current figures. Note that the suite's 99.9% and the article's 99.98% differ only by denominator — the suite divides by a rounded 23,400,000, the article by the registry base 23,373,283.
• The Taiwanese figures — recomputed directly from cached/zh_TW/lastname_male.wgt on 18 July 2026: 2,707 entries, weights summing to 23,367,536, top ten summing to 12,339,778, giving 52.8074% of the corpus. Against the full registry base of 23,373,283 the same top ten is 52.7944%. The ministry publishes 52.79% over 2,731 surnames. The 5,747-bearer difference between the two denominators is exactly the 24 excluded rows: 其他 at 5,174 bearers plus 23 Private Use Area rows. The two shares are therefore two different measurements, not a contradiction.