Your first audit, in detail

This walks one real corral certify --local run start to finish, using the same password-validator example published in the field note “Terribly sorry to trouble you… but your tests, um, well. They suck, really”. Nothing here is staged for the docs — it’s the actual shape of a verdict this gate produced.

The code under review

Elastic-2.0

package passwd

import "unicode"

// Valid reports whether p is a valid password: length >= 12 AND it contains an
// uppercase letter, a lowercase letter, a digit, and a symbol.
func Valid(p string) bool {
  if len(p) < 12 {
    return false
  }
  var up, lo, di, sy bool
  for _, r := range p {
    switch {
    case unicode.IsUpper(r):
      up = true
    case unicode.IsLower(r):
      lo = true
    case unicode.IsDigit(r):
      di = true
    case unicode.IsPunct(r) || unicode.IsSymbol(r):
      sy = true
    }
  }
  return up && lo && di && sy
}

The developer’s own test

A test that only ever feeds a valid password:

func TestGappy_ValidLength(t *testing.T) {
  if !Valid("Abcdefgh1!xy") {
    t.Fatal("a 12-char password was rejected")
  }
}
func TestGappy_TooShort(t *testing.T) {
  if Valid("Ab1!xyz") {
    t.Fatal("a 7-char password was accepted")
  }
}

Green in CI. Looks reasonable — it checks a boundary (7 chars vs 12) and a happy-path accept.

Running it

export ANTHROPIC_API_KEY=sk-ant-...

corral certify --local \
  --code passwd.go \
  --test passwd_gappy_test.go \
  --writer-model claude-sonnet-5 \
  --mutant-model claude-sonnet-5 \
  --critic-model claude-haiku-4-5 \
  --goal "Valid must require length >= 12, an uppercase letter, a lowercase letter, a digit, and a symbol" \
  --out verdict.json \
  -- go test ./...

The model names above are an example, not a default — corral has none. Every seat is yours to name, from whichever provider you hold a key for; the only rule is that the test-critic must differ from the test-writer.

--code/--test may be absolute or relative paths. On Ubuntu 24.04, if bwrap won’t start (apparmor disables unprivileged user namespaces by default), add --jail container and export CORRALAI_EXEC_IMAGE=golang:1.26 to run the jailed check in Docker instead — no sudo required.

What happens inside the jail

1. Preflight. The Go toolchain is checked (Preflight() fails closed if go isn’t on the jail-visible PATH), and the sandbox backend is resolved — bwrap by default on Linux, refusing to fall back to unsandboxed if it can’t start.
2. Mutants. A mutant-generator model seeds goal-violating mutants into Valid — dropping the digit check, weakening the length bound, flipping the symbol test, and so on. On a file with several functions the generation fans out: one seat per group of functions (balanced by complexity), so every function is probed rather than just the one a single generator picked — --max-shards bounds the width, and it auto-sizes to your machine otherwise. A mutant that makes the suite hang (a broken loop bound, say) is killed fast and counted as caught, so a non-terminating mutant can’t stall the run.
3. Dev kill-rate. The existing test above is run, in the jail, against every mutant. It only ever calls Valid with a password that already satisfies every rule (or is simply too short) — so a mutant that loosens a rule (say, dropping the digit requirement) still passes: the input never exercised the rule it broke. Several mutants survive — a survivor is a planted mutant your suite didn’t catch. corral discloses it unadjudicated: it may be a real untested edge, or an equivalent mutant (a change nothing can observe, so no test could ever catch it). Which one is the human’s call, not the tool’s — corral won’t call an unproven survivor a real bug.
4. Test-writer. For each surviving mutant, a test-writer model (a different model than the critic, enforced — decorrelation isn’t optional) writes a targeted test; if its first attempt doesn’t compile, the compiler’s own error is fed back so it corrects rather than blindly repeating. When the jail confirms the test kills the mutant, that survivor is now a proven, catchable gap — not a fluke of the harness — and the test is handed back for you to add.
5. Test-critic. A decorrelated critic model reads the dev’s test file cold and flags what it would have said in review: the suite never tries an invalid password that’s the right length. Its findings are carried as an unverified second opinion — advisory, never gating the signed verdict, which is execution-proven only.
6. Shadow challenger (only if you name a --shadow-model). In parallel, a second model attacks the same regions — a same-ground, head-to-head measurement of one model against another. It’s recorded for comparison and never enters the verdict; the exam’s difficulty is set by the primary model alone, so certification means exactly what it did before.

Whatever happens, the run converges to a signed verdict — even if the herd can’t author a compiling test to kill a survivor, it routes to needs-review and hands back what it found rather than spinning.

The verdict

dev_kill_rate = 0.75   (killed 6 of 8 mutants, 2 survived)
status: needs-review   (below the 0.8 certify bar)

Below the bar, so the run routes to a human instead of auto-certifying — the honest failure mode. The kill-rate is measured, not asserted: it’s mutation testing executed in the sandbox, never the model’s self-report of how good the test is.

The signed record

The verdict is written to a local, tamper-evident ledger (the same certify chain corral certify uses for the non---local path) and printed to stdout — subject (repo/commit), the models that played each role, the kill-rate, and the status. Because you passed --out, the signed record is also written as a self-contained file, which re-verifies any time, fully offline (the run prints this line for you, with your key filled in):

corral certify verify verdict.json --pubkey "$(corral certify pubkey)" --allow-unanchored

--allow-unanchored is required: a --local record is signed by your own key but never submitted to a public transparency log, so verify makes you opt in to that weaker “signed-by-you, not third-party-witnessed” trust explicitly.

Try it on your own code

Point --code and --test at a real file and its test, describe the guarantee in --goal, and pass whatever check command actually exercises it (go test ./..., pytest, npm test, …). See Getting started for the jail-visibility gotcha (system-installed toolchain, not a --user/snap install) and the Ubuntu-apparmor fix if bwrap won’t start.

Reproduce a published audit yourself

The two sportspicker-core tapes on the recordings page are the easiest published runs to reproduce, because the subject is a public, zero-dependency library and the audit needs nothing but pytest. It scores sports pick’em contests, and the guarantee it has to defend is one anyone can hold in their head: a contest contributes exactly its budget — whatever scoring rule is used, however many people played, the points awarded in a round sum to that round’s pot.

git clone https://github.com/pdbethke/sportspicker-core && cd sportspicker-core
python3 -m venv .venv && .venv/bin/pip install pytest

corral certify --local --repo-dir . \
  --code sportspicker_core/awards.py --test tests_core/test_awards.py \
  --goal "A contest contributes exactly its budget: for any round, the points awarded across the field sum to that round's pot — whatever normalization strategy is chosen, however many members played, and however they scored, including a round nobody scored in. min_participants gates on members who SCORED, not members who merely submitted picks." \
  --mutant-model gemini-3.6-flash --writer-model gemini-3.6-flash \
  --critic-model claude-haiku-4-5-20251001 \
  -- .venv/bin/python -m pytest tests_core -q

If you want to see what the library actually does before auditing it, there is a command line over the same arithmetic in sportspicker-cli — one dependency, which is the library itself:

sportspicker award round.json --budget 1000 --rounds 4
#   pot 250.0000   awarded 250.0000

Change the strategy and the split changes while that total does not. That is the guarantee the audit is grading, visible in one command.

main scored 18/20 — CERTIFIED. Then run the identical command on git checkout demo/bug-counts-pickers, a branch that carries a real bug and a green 80-test suite: 15/20 — NEEDS-REVIEW. Same file, same command, and CI is happy with both. That gap is the whole product in one command.

Your numbers will not match ours exactly — the mutants are generated fresh each run, so the two verdicts are samples, not constants. What reproduces is the direction: the thinner suite scores materially worse on the same code.