Mutants, survivors, kill rates

corral’s output is full of words like mutant, survivor and kill rate. They’re borrowed from an established field, and if you haven’t met them before the reports read like comic-book solicitations. This page is the vocabulary, plus a straight answer to the fair question “isn’t this just mutation testing?”

Short version: neither mutation testing nor generating tests for survivors is new. Corral packages those techniques with execution checks, selectable model seats, CI integration, and a portable audit record.

A mutant is a deliberately broken copy of your code

Not a superpower. A mutation in the biological sense — a small, specific change to the original.

Say this is your code:

def is_adult(age):
    return age >= 18

A mutant is that function with one thing wrong on purpose:

def is_adult(age):
    return age > 18      # >= became >

Now run your existing test suite against the broken version and watch what happens:

• If a test fails, your suite noticed. The mutant is killed.
• If every test still passes, your suite did not detect the edit. The mutant survived.

A survivor is a candidate test gap, not automatically a behavior change or a bug. The edit may be equivalent to the original program. For a non-equivalent edit, the suite passed despite that difference in this run.

That’s the whole trick, and it’s why the numbers mean something a coverage percentage doesn’t. Coverage tells you a line ran. A killed mutant tells you a line ran and something was actually checking the result. A line can have 100% coverage and kill nothing.

The rest of the vocabulary

Kill rate — of the mutants actually graded, the fraction your suite caught. Invalid and unmeasured mutants are excluded. This is a measurement of this exam, not a general correctness grade. It is measured by executing your tests, never self-reported by a model.

Baseline — a run of your suite against your unmodified code, before any mutants. It has to pass. If your suite is red to begin with, every subsequent result is meaningless, so corral refuses to grade and tells you what your suite said. See Limitations.

Canary — a deliberately invalid version of the file, broken so badly it cannot compile or import. Your suite must fail on it. If it passes, your tests provably never load the file under audit, so any kill rate would be a number about nothing. This catches the embarrassing case where the tests never exercised the target at all.

Survivor, disclosed but unadjudicated — corral does not call a survivor a bug. A survivor is a candidate: a change your tests didn’t catch. It stays disclosed and unproven until a test actually kills it by execution.

Proven missed — a survivor that corral then wrote a test for, where that test compiled, ran, passed on your unmodified code, and failed on the mutant. It is an execution-demonstrated test gap, not a defect found in the original program. The test still needs review for intended behavior, relevance and maintainability. The writer may fail to produce such a test; an unproven survivor stays unproven.

What the confidence interval does and does not mean

The reported 95% Wilson interval uses a binomial sampling model. Its usual interpretation assumes independent observations from a representative sample of a defined fault population. Generated faults can be correlated, goal-directed and influenced by earlier runs; distinct edits are not necessarily independent samples. Corral’s exam-size gate uses distinct graded edits rather than counting duplicate edits as extra observations, but that does not establish representativeness.

The interval is not a probability that your code is correct, and a narrow interval is not proof that the exam covered every important failure mode. Certification applies to the recorded exam and execution conditions.

A recorded fault/test pair remains inspectable, but a fresh run can discover different gaps, or none, even on unchanged code. Both kill-rate and proven-gap gates can vary. Replay a recorded mutant set when comparing test changes; test generation and flaky execution can still vary even with that set held fixed.

Individually proven tests are not automatically a merge-ready bundle

Run generated tests in your full suite and review their assertions before adding them. Tests that pass alone can interfere through shared state, fixtures or order. On unreleased main, after rc.24, a per-survivor run with multiple proven tests also runs the merged authored file once on the unmodified code. Its authored_together result is pass, fail, or absent when not checked. A failure produces a warning; it does not change the individual proven count or certification status. A pass is one run, not a guarantee against flakiness.

This technique is about fifty years old

Worth saying plainly, because it’s true and because you’d find out anyway.

Mutation testing goes back to the 1970s. Richard Lipton proposed the idea as a student around 1971; the foundational papers are Hamlet’s Testing Programs with the Aid of a Compiler (1977) and DeMillo, Lipton and Sayward’s Hints on Test Data Selection (1978). Working tools existed by the late 1980s. There are mature implementations for most ecosystems today — PIT for Java, Stryker for JavaScript/TypeScript, mutmut and cosmic-ray for Python, several for Go and Ruby.

If you already use one of those, you already understand corral’s measurement half, and you should keep using them — a dedicated mutation-testing tool for your language will plant more mutants, faster, than corral does.

So why didn’t it take over the world? Two reasons that have dogged it since the beginning:

It’s expensive. You run the whole test suite once per mutant. Forty mutants against a two-minute suite is eighty minutes for one file. That cost is the reason mutation testing lives in research papers and specialist CI jobs rather than in everyone’s pipeline, and it is still the honest constraint on corral — see Limitations.

Equivalent mutants. Some mutations don’t actually change behaviour. Swapping < for <= on a bound that can never be hit produces a mutant no test could ever kill, because there is nothing to catch. Classically these inflate your survivor count with noise, and deciding which survivors are equivalent is undecidable in general. corral’s answer is procedural rather than clever: a survivor is never reported as a bug, only as a candidate, until a test kills it by execution. An equivalent mutant simply never gets proven, and never gets claimed.

Related work and Corral’s integration

Mutation-guided test generation has substantial prior work. ACH, described in Mutation-Guided LLM-based Test Generation at Meta (Foster et al., arXiv, January 2025; FSE 2025 Companion, Industry track), plants concern-directed faults and generates tests that kill them. LLMutantKiller (Khalili, Domondon, Garg and Tip; ISSTA 2026) generates tests for surviving mutants produced by Stryker rather than by an LLM. Corral did not invent the plant-then-kill idea, and these citations are not a head-to-head performance comparison. Corral’s distinction is the integrated workflow and how it handles evidence: grading only by execution in a sandbox, a hash-linked ledger, and a decorrelated critic that is advisory only.

Corral’s offering is the integration of the following capabilities:

The faults are proposed against a stated goal, not by mechanical operator, as ACH also does. Classical mutation testing flips syntax: every + becomes -, every < becomes <=. That’s cheap and thorough and produces a lot of uninteresting damage. corral asks a model to introduce violations of a specific correctness or security goal — the kind of mistake a tired human would plausibly make. Fewer mutants, aimed at behaviour that matters.

Tests target survivors. Corral takes the survivors and asks a test-writer model for tests targeting them, then executes the results. (The writer may be the same model that planted the mutants; what decides the outcome is the run, not either model.) A proven test must compile, pass on your unmodified code, and fail on the mutant. Only then does the gap count as proven — and you get the test, not just the diagnosis.

The result carries evidence. Local certification records are signed; repository and review ledger entries are signed when a certify key is configured and otherwise remain unsigned and hash-linked. The record can be queried later; where a witness is configured, a signed statement can also be anchored to an external transparency log (Sigstore Rekor) via --transparency — shipped, opt-in, off by default, and never on the --local path most of this site’s examples run. See Verify a record yourself for the difference between a signed record and an anchored one. It’s built to gate a merge, which means it has to be something a skeptical third party can check rather than something you tell them.

The value to evaluate is whether this integration returns useful, reviewable tests at an acceptable setup and execution cost. Establishing that advantage requires comparative runs and independent maintainer acceptance, not a novelty claim.

Reading a real verdict

Putting it together, a line like this:

src/flask/cli.py: kill rate 0.47 (21 survivor(s), 12 proven missed)

says: corral planted faults in cli.py and ran flask’s own tests against each one. The suite caught 47% of them. 21 got through unnoticed. For 12 of those 21, corral wrote a test that demonstrably catches the fault — so those are not opinions about test quality, they’re twelve specific gaps with twelve specific tests.

The other 9 remain survivors: disclosed, unproven, and never called bugs.