Real audits, recorded. Each is corral certifying a change by execution: a decorrelated cross-vendor herd plants faults in the code, checks whether the developer's own tests catch them, and signs a tamper-evident verdict — no one judging their own cause. Some clear the bar and certify; some leave too many survivors the suite didn't kill and are sent back — corral discloses those for a human to judge, it doesn't rule them defects. Both are here, honestly — the gate showing its work either way.
Every one was exported through the same deny-list + human-manifest privacy gate as the landing hero, and is offline-verifiable from its signed record. Open the tests tab in any replay to see the verdict, the code under review with the surviving fault highlighted, and the suite it graded. Pick a card to replay it on the corral canvas.
Certify internal/certify/certify.go — corral’s own signing code, the
package that builds the hash-linked ledger, wraps the head in an in-toto/SLSA
attestation, and signs/verifies it with Ed25519 — against corral’s own Go test
suite, by execution. No special-casing: the same --repo-dir path any other
target gets, pointed at ourselves. Nemo iudex in causa sua only means
something if the tool submits to it too.
Gemini 3.6 Flash planted 78 goal-violating mutants across the file, sharded across generator seats. The package’s own suite, run in the jail against every one, killed 43 of 78 — a 55% dev kill-rate. 35 survived. That’s under the certify bar, so the gate returned NEEDS-REVIEW and signed the verdict anyway — a disclosed result, not a hidden one.
The decorrelated critic here is Gemini 3.7 Flash — a different model than the one that planted the faults and wrote the exposing tests, reading the suite cold. Its findings are advisory only; they never gate the signed verdict.
Watch the tape closely mid-run and you can see the pool’s own corrective loop
firing on live signing code: the writer’s authored test for survivor s0/m5
compiled but failed on the clean, unmutated file — reissued once with the
failure fed back rather than accepted. A separate writer seat for s3/m10
didn’t compile at all on the first attempt and was reissued too. Both are
ordinary parts of the per-survivor proving loop, not something dressed up for
this recording — the same retry-with-feedback mechanic every tape in this
gallery relies on, just visible here because it happened on the code that
implements it.
Where it landed matters: the writer went on to author compiling, clean-passing tests for 30 of the 35 survivors, proving them catchable by execution rather than leaving them as unverified claims. Five survivors are still open — either real untested edges in our own signing path, or equivalent mutants nothing can catch, and corral doesn’t adjudicate the difference itself.
We could have picked a flattering file. We picked our own signing code, ran the same gate we ship, and published NEEDS-REVIEW with 35 survivors sitting in public view — not because the code is known-broken, but because the point of a decorrelated gate evaporates the moment its maker gets to pick which of its own results to show. Open the tests tab to see the surviving faults against the code the suite passed anyway, and the corrective retry firing in real time.
Certify version4.go from google/uuid — one of the most-used Go libraries
there is — against the package’s own test suite, by execution. The goal: New /
NewRandom return a valid RFC-4122 version-4 UUID (122 random bits, the version
nibble set to 4, the variant bits set to RFC 4122).
Claude Sonnet 5 planted 20 goal-violating mutants across the file (sharded four ways); google/uuid’s own suite, run in the jail against every one, killed 13 of 20 — a 65% kill-rate, and 7 survived. The gate returned NEEDS-REVIEW and signed the verdict. This is the point worth sitting with: a library this widely trusted, with a real test suite, still leaves a third of the planted faults uncaught.
The decorrelated critic (Haiku 4.5) put its finger on the shape of the gap:
TestRandomUUID calls New() many times and asserts each result has version 4
and the RFC-4122 variant — but it never verifies the 122 bits are actually
random. So a mutant that fixes or narrows the randomness (while keeping the
version/variant bytes correct) sails straight past. The suite checks the
structure of a UUID thoroughly and its randomness not at all. (The critic’s
read is unverified advice, marked as such; the 65% is what the jail measured.)
It’s the “your tests suck” thesis on code nobody would call badly tested. corral doesn’t grade google/uuid as bad — it grades it by execution and hands back exactly which faults its suite can’t see. Open the tests tab to watch a surviving mutant highlighted against the code the suite passed anyway.
Certify a file from more-itertools — more_itertools/recipes.py, 67
functions we didn’t write, from a well-loved zero-dependency library — against the
library’s own test suite (tests/test_recipes.py), by execution. The
generation fanned out across shards so every function got probed, not just
whichever one a single generator happened to pick.
Gemini 3.5 Flash planted 20 goal-violating mutants across the file; the library’s own suite, run in the jail against every one, killed 18 of 20 — a dev kill-rate of 90%, measured by execution, not asserted. That cleared the bar (0.8), so the gate returned CERTIFIED and signed the verdict — offline-verifiable from the record.
Two of the twenty mutants survived — the suite passed despite them. Corral records both rather than rounding up to “spotless,” but it makes no claim that they’re defects. A survivor is either a real untested edge or an equivalent mutant — an injected change with no observable effect, which no test can catch by construction — and corral only calls a survivor a proven gap when the pool authors a compiling test that actually kills it (it didn’t here). So this is not “more-itertools missed two tests”: it’s the tool catching 90% of injected faults against a genuinely strong suite, and disclosing the two it couldn’t kill, unadjudicated, for a human to judge. Open the tests tab to see them highlighted against the code the suite passed anyway.
The decorrelated test-critic here is a stronger, different-tier model — Gemini 3.1 Pro — reading the suite cold while a lighter model did the planting and writing. Its job is to flag tests that don’t actually test anything. It flagged one, and — the point — it holds up under execution:
test_full_permutation, the if i == r “not permuted” check is dead:
i is a range object and r is a tuple, and in Python 3 range == tuple
is always False. The check can never fire. The test still has a real
assertEqual around it, so it isn’t vacuous — but that one guard is inert, and
the critic scoped its wording precisely to the check. Accurate, not overstated.That precision is the point. An earlier run of this same file, with a lighter
same-vendor critic, produced a confident hallucination — it flagged
test_negative_take as vacuous, insisting islice silently swallows a negative
count. It doesn’t: take(-3, …) raises ValueError, so the test passes for
exactly the right reason. A stronger, decorrelated critic makes no such mistake.
Either way, corral treats the critic’s word as what it is: unverified advice that never gates the signed verdict — only a jail and an exit code certify. But watching a more reliable model produce more reliable advice is the whole helper turn: better decorrelation makes your tests stronger from both ends, and every claim on this tape is one you can re-run yourself.
Certify src/index.ts from vercel/ms — the duration parser/formatter
underneath a huge share of the Node ecosystem’s timeouts and cache TTLs —
against the project’s own jest suite (167 tests), by execution, in the jail.
The recording that ships here is the second attempt. The first came back
COULD-NOT-GRADE: the dev suite didn’t even pass on the clean, unmutated
code inside the jail. The failure wasn’t a real bug in ms — jest’s own
loader threw Cannot find module 'import-local' trying to launch
jest/bin/jest.js inside the sandboxed copy. The jail had bound
node_modules in read-only, exactly as the log records — but a bound
directory of files isn’t the same guarantee for every install layout. Some
package managers (pnpm’s content-addressable store is the sharpest example)
install node_modules as a tree of symlinks pointing outside that
directory, into a global store; bind-mount the tree without the store behind
it and a symlink resolves to nothing, however complete the top-level listing
looks. Corral called that what it was — a build/environment failure, not a
test-quality verdict — and graded zero of the 34 mutants it had already
generated rather than pretend a broken baseline proved anything.
With the dependency layout the jail could actually resolve, Gemini 3.6 Flash planted 34 goal-violating mutants across 7 sharded regions of the file. The project’s own 167-test jest suite, run against every one, killed 32 of 34 — a 94% dev kill-rate, clearing certify. The gate returned CERTIFIED and signed the verdict.
2 survivors remained. The per-survivor writer fanned out, hit one
tsc --strict compile failure on the first pass (a relative import missing
its explicit extension under nodenext module resolution — reissued and
fixed), then had one authored test fail on the clean baseline and get
reissued again with the failure fed back. On the third pass, both survivors
came back proven catchable by execution — not asserted, not left as
0 of 2.
The lesson isn’t “vercel/ms has a bug” — it doesn’t, by this measurement.
It’s that a mutation jail’s isolation guarantee is only as good as what it
actually bind-mounts, and a symlink-based install can look complete on ls
while resolving to nothing inside the sandbox. If a “clean” baseline fails
inside the jail with a module-resolution error and passes fine on the host,
check what your package manager actually put on disk before assuming the
target project is broken — corral’s own COULD-NOT-GRADE status exists
precisely so that question gets asked instead of silently swallowed into a
wrong verdict.
Certify a password validator — valid iff length ≥ 12 and it contains an uppercase letter, a lowercase letter, a digit, and a symbol — against a Python suite that only ever feeds one valid password (and one too-short one), so it never exercises the four character-class rules.
Claude Sonnet 5 planted 5 goal-violating mutants; the developer’s own suite, run in the jail against every one, killed 0 of 5. Every survivor is a dropped character-class check the length-only test can’t see. A test-writer (Sonnet 5) then authored a test that killed all 5 survivors — proving the gaps are real and catchable, not equivalent mutants. The gate returned NEEDS-REVIEW and signed the verdict: it will not certify a suite that guards a fraction of the spec. The decorrelated critic (Haiku 4.5) independently flagged both tests as vacuous — and here it was right: neither can fail on a character-class mutation, and that’s exactly what the execution showed.
The same blind spot as the Go password recording — a length-only test that sails past every character-class fault — now in Python, from corral’s own known-adequacy eval corpus. It’s the whole loop in about forty seconds: find the gap by execution (a 0% kill-rate, measured, not asserted), prove it (a written test that kills every survivor), and grade the suite honestly (needs-review, signed). Open the tests tab to watch a surviving fault highlighted against the code the suite passed anyway.
Certify a change to internal/fence/fence.go — the wrapper that fences
untrusted content (ingested documents, agent-written memory, reported findings)
so a consuming agent can’t mistake it for its authoritative task — by
execution. The question isn’t “does the code look right”; it’s “does this
file’s own test suite actually guard the security property: that untrusted
content can never forge or close the fence?”
This is the audit gate, not a builder — nemo iudex in causa sua, no one is judge in their own cause. A decorrelated cross-vendor herd put the suite to the test: Claude Sonnet 5 planted the faults (and drafted the exposing test), and a different model — Gemini 3.5 Flash — graded whether the developer’s tests caught them. The model that plants the exposing fault is never the model that judges the suite. That separation is enforced, not hoped.
Open the tests tab (the audit lens) to see the whole thing in one place: the signed verdict, the code under review with the surviving fault highlighted, and the developer’s suite corral graded. Every catch on this tape is proven by a test that actually ran in the jail and killed a fault — never a model’s word.
corral certify verify). You don’t trust it; you re-run it.corral certify is the shipped atom; the adversarial pool that produced this
tape is experimental and off by default. And one run is a data point, not a
ranking — fence.go’s suite is genuinely strong, so whether a fault survives
is partly a coin flip. The bug-catching numbers only mean something at volume,
which the eval harness accrues across a whole corpus before any of it gets
published. We’ll always tell you which floor you’re standing on.
Certify lib/text/levenshtein.rb from threedaymonk/text — a real, pure-Ruby
edit-distance implementation (UTF-8 aware) with a minitest suite — against its
own tests, by execution. The goal: distance(a, b) returns the minimum number of
single-character insertions, deletions, or substitutions to turn a into b.
Run head-on, the suite looks pristine: 35 tests, 5,241 assertions, 100% passing. Then corral graded it by execution. Claude Sonnet 5 planted 5 goal-violating mutants in the algorithm; the library’s own minitest suite, run in the jail against each, killed only 3 of 5 — a 60% kill-rate with 2 survivors. That is below the bar (0.8), so the gate returned NEEDS-REVIEW and handed the two surviving faults back.
This is the gap the whole tool exists to expose: a green, assertion-heavy suite that still lets two goal-violating edits through undetected. Passing count is not adequacy — and corral says so by execution rather than by vibe. Open the tests tab to see the two survivors highlighted against the code the suite passed anyway.
The decorrelated critic (Haiku 4.5), reading the suite cold, flagged something
tangential but telling: a helper in the test file itself
(LevenshteinGeneratedDataTest#substitute) references an undefined local — se
where seq was meant — which raises a NameError on the path that hits it. It’s
the critic noticing a latent break inside the tests that grade the code. That
finding is advisory, marked unverified, and never part of the signed 60% —
only the jailed kill-rate certifies — but it’s a second, independent reason to
look twice at this suite.
Same loop as the Go and Python audits, in a third language (Ruby, minitest): plant faults, run the dev’s own suite in a jail, grade by the kill-rate — no self-report. Here the honest answer was not yet — and that refusal, on a suite that passes 100% of its own assertions, is the point.
Certify lib/text/levenshtein.rb from threedaymonk/text — a small,
zero-dependency UTF-8 edit-distance implementation we didn’t write — against
the gem’s own minitest suite, run through rake, by execution.
The first attempt against this file never spent a token. The operator’s test
command named a single spec file directly, and corral writes its killing test
beside the developer’s own tests — test/levenshtein_corral_test.rb — which
a single-file command will never collect. Corral checked that before running
any model: your command does not collect that file, so this audit could not
prove a gap even if it found one. It refused, in 2.5 seconds, for free, and
told the operator exactly what to widen the command to (a directory or the
runner’s own discovery, not one spec path). Nothing was spent on a verdict
that couldn’t have meant anything.
With the test command widened, Gemini 3.6 Flash planted 9 goal-violating mutants across the file’s three functions. The gem’s own suite, run in the jail against every one, killed 7 of 9 — a dev kill-rate of about 78%. That left the gate short of certify, and it returned NEEDS-REVIEW, signed.
2 survivors remained. The per-survivor writer authored a compiling test for each and proved 1 of the 2 catchable by execution — a real gap in the gem’s own suite, not a hypothetical one. The second stayed unresolved: either a genuine untested edge the writer couldn’t pin down in the run, or an equivalent mutant with no observable effect. Corral discloses it rather than rounding it away, and doesn’t call it a defect on its own authority.
Two lessons, not one. First: the pre-spend check is real money-in-the-bank — a misconfigured command gets refused before a single model call, not after a paid run produces a verdict that was structurally incapable of proving anything. Second: “1 proven” is not “1 of 2, so half-credit.” It means one survivor now has a compiling test in hand that kills it, and one is still open and honestly labeled that way — the two are not interchangeable claims, and this tape is the difference in one screen.
Certify sportspicker_core/awards.py — the award arithmetic for a pick’em
scoring library we wrote — against its own suite (90 tests, green), by
execution. Same file this gallery has shown before, this time on main, not
the deliberately broken demo branch.
Gemini 3.6 Flash planted 20 goal-violating mutants across the file. The library’s own suite, run in the jail against every one, killed 18 of 20 — a 90% dev kill-rate, clearing the 0.8 bar. The gate returned CERTIFIED and signed the verdict.
2 survivors remained — the suite passed despite them. Corral doesn’t stop at the pass/fail line: the per-survivor writer went back in and authored a compiling test for each one, and both proved catchable by execution — the decorrelated Gemini 3.7 Flash critic then read the resulting suite and flagged nothing further. So the honest shape of this run is not “certified, done” — it’s certified and both of the two gaps a green 90-test suite still had were found and proven, not just counted.
CERTIFIED is not the same claim as “nothing left to find.” A 90% kill-rate clears the bar and a verdict gets signed — and the tool kept going anyway, proving out the two survivors a merge-worthy suite still missed. That’s the whole differentiator from a green CI check: passing isn’t the end of the measurement, it’s where this one starts.
Certify awards.py from sportspicker-core — a small, zero-dependency
library that scores sports pick’em contests — against the library’s own test
suite, by execution. The goal it had to defend is the guarantee the whole design
rests on: a contest contributes exactly its budget. Whatever scoring rule is
chosen, however many members played, and however they scored, the points awarded
in a round must sum to that round’s pot.
The branch under audit is deliberately broken, and says so in its own
DEMO-BUG.md. min_participants counts everyone who submitted picks rather
than everyone who scored, and the test that would have caught it is gone —
the remaining cases all use fields where every member scored, so the two counts
are the same number and the distinction never arises. The suite is green: 80
tests, well under a second. CI has nothing to say about this branch.
Gemini 3.6 Flash planted 20 goal-violating mutants across the file, sharded four ways by function. The library’s own suite, run in the jail against every one, killed 15 of 20 — a 75% kill-rate — and 5 survived. The gate returned NEEDS-REVIEW and signed the verdict.
Where the survivors landed is the interesting part. Two sit in effective_pot,
the function that carries the planted bug; two in award_round; one in
pot_for. Among them:
positions = competition_ranks(scores) replaced by a naive enumerate — so
tied members stop sharing a rank. It survives because no remaining fixture
contains a tie.min_participants defaulting to 1 instead of 0.scoring >= minimum boundary flipped to >, so a field sitting exactly
on the minimum is treated as short.Be precise about what this is and isn’t. corral grades test adequacy by
execution; it did not announce “there is a planted bug here”, and it does not
claim to. What it did is measure that the tests guarding this file no longer
defend it — and the same file on main, one test richer, scores 0.90 and
certifies. Same code path, same command, two verdicts, both with a green suite.
The decorrelated critic (Claude Haiku 4.5) flagged three tests it read as vacuous, including an assertion that a zero-scoring member is awarded zero points — which holds arithmetically no matter what the distribution logic does. That is a second model’s opinion, marked UNVERIFIED, and it never gates the verdict. The 75% is what the jail measured.
It is the thesis in one screen. The tests pass. The branch would merge. The bug is real, documented, and sitting in the diff — and the only instrument that registers anything is the one that runs the suite against faults it planted itself. Open the tests tab and watch a surviving mutant highlighted against the code the suite passed anyway.
The same question as the sportspicker branch audit, against the same file — but
on main, with the full suite. Certify awards.py from
sportspicker-core against its own 81 tests, defending the guarantee that a
contest contributes exactly its budget.
Gemini 3.6 Flash planted 20 mutants across the file’s four functions. The suite killed 18 of 20 — a 90% kill-rate — and the gate returned CERTIFIED.
The two survivors are both off-by-one boundary conditions, and both are real:
pot_for: if round_count <= 0 widened to <= 1, so a single-round
contest pays nothing at all — a direct violation of the budget guarantee,
in the one shape (a one-off fight card) the library exists to support.effective_pot: if scoring >= minimum narrowed to >, so a field landing
exactly on the minimum turnout is treated as short of it.The test-writer then proved both catchable by execution and handed back a test that kills them. The critic flagged no vacuous tests.
It is the honest companion to the branch audit — and a caution about self-graded numbers. This repository ships its own mutation dry-run: sixteen goal-violating changes planted by hand, and the suite kills all sixteen. The README says to treat 16/16 as a floor rather than a claim, because those are the failures the test author imagined.
That caveat turns out to be the whole point. An adversarial generator with no stake in the author’s assumptions planted two faults he had not imagined, and his suite let both through — one of which silently zeroes out every one-round contest. A 90% CERTIFIED verdict is not a gold star; it is a measurement, with the two gaps named and a killing test attached.
Both survivors were real, and both are now covered. The library added a test
pinning pot_for(budget, 1) to the full budget, and another pinning a field
landing exactly on min_participants — asserted in void mode, because the
existing test at that boundary ran in scale mode, where pot * (scoring / minimum) equals pot and the right answer comes back under either comparison.
It read as coverage and discriminated nothing.
The critic earned its keep on the re-run too. It flagged a test asserting that a zero-scoring member receives zero points — which holds arithmetically whatever the code does, since normalized weights already give a non-scorer nothing. That member now scores, so the assertion has to be earned.
A re-audit of the fixed suite scored 0.95, CERTIFIED, with one survivor.
Read the improvement carefully, because this is exactly where a kill-rate can be oversold: mutants are generated fresh each run, so 0.90 → 0.95 is two samples, not a controlled measurement. What is not a sample is that two specific faults got through before and are now covered by tests checked against those faults directly — apply the mutant, watch that test and only that test fail.
The tape above is left as it was recorded. It is a dated artifact of the commit
it graded, and demo/thin-boundaries in the repository preserves that exact
suite so the gaps stay readable rather than disappearing into a diff.
Certify src/index.ts from vercel/ms — the tiny, ubiquitous
duration-string library (ms('2 days') → 172800000, and back) — against its
own Jest suite, by execution. This is the first recording of a real
TypeScript project audited with its real, dependency-heavy test toolchain.
ms’s test run pulls in a 337 MB node_modules (Jest, its TS transform, the
lot). corral’s jail seeds a copied workspace with a size cap, and 337 MB blows
straight through it — so until recently this audit failed at the seed stage,
before a single test ran. The fix, shipped just before this run: corral
bind-mounts dependency directories read-only into the jail instead of copying
them. You can see it in the run’s first line — deps: bound 1 dir(s) read-only (node_modules). Deps must be present (vendored, exactly as CI installs them);
corral binds them, never installs them. That’s what lets it audit real-world
JS/TS code and not just zero-dependency toys.
Claude Sonnet 5 planted 5 mutants; ms’s own Jest suite killed 4 of 5 — 80%, 1 survivor, NEEDS-REVIEW, signed.
Getting an honest number here took one correction worth telling on ourselves. corral certifies by exit code — it runs your test command and trusts what the process returns. ms’s Jest config enforces a 100% coverage threshold, so the command exits non-zero even when every test passes (coverage sits at 98.4%). On the first run that made the exit code meaningless: every mutant “failed” for the coverage gate, not for catching a bug, and the kill-rate signal was garbage. Re-running with coverage off — so the exit code reflects test pass/fail — gave the real 80%. The takeaway is a genuine edge of execution-gated auditing: a suite whose exit status is dominated by a coverage (or lint) gate confounds a tool that reads exit codes. corral is honest about what it can and can’t infer from a process’s return.
Real TypeScript, a real Jest suite, real vendored dependencies — the exact shape of code the earlier toy recordings couldn’t prove out — audited by execution to a signed verdict. Open the tests tab to see the surviving fault against the code ms’s suite passed anyway.
The cross-recording metric for audits isn't a findings table — it's thebug-catching scorecard: which model actually catches bugs,proven by execution, per (model, role). It accrues as audits run — a cell stays provisional until it has enough of them, so we don't publish a confident number off a handful — and lives atcorral scorecard / /api/bugcatch, offline-verifiable from each signed record. Fugu's routing scores come out of a trained black box; this comes out of a test that really ran and really killed a fault.