Field notes
Thinking out loud from the corral
Notes from the corral: where we think out loud about multi-agent orchestration, model composition, and the fences that make it safe.
Half a million tokens to spot a one-line change
The first time our cost meter ran, it printed one line about our own tool that nobody had ever been able to see: the test-writing model was being handed the same 36 KB file twenty-four times per call. What happened in the twelve hours after that line, and the rule it left behind: cold exam, warm prover.
Read the note →The database found the bugs
We made our audit tool write every decision — every planted bug's fate, every model call's cost, every downgrade's reason — into a DuckDB file. Within 48 hours the database had caught five bugs in the tool itself that no test suite had. A field note on analytics as a debugger, and why the receipts now live in a warehouse you own.
Read the note →Local models break things well and test them badly
We ran the same 9B model in two seats of the same audit. As the bug-planter it beat a human test suite thirteen times. As the test-writer it could not produce a file that compiled. That asymmetry decides where a local model belongs — and it happens to put the free one in the expensive seat.
Read the note →The gate is whatever you happened to install
Three times this week we found corral crediting a test suite with catching a bug it never detected. Fixing the third one exposed the shape underneath: the check that decides whether a mutant counts at all depends on what is installed on the machine — so the same audit can report a different denominator on two boxes.
Read the note →Eighty-eight tests that were not there
We pointed corral at three third-party Go libraries with good reputations and real test suites. It planted bugs their tests missed, then wrote and RAN tests that catch them — 88 of those, across 15 files. Here is the whole result, including the third of the panel it could not grade at all.
Read the note →Both models missed the same nine
Corral's design document has claimed a decorrelated herd for months. Nothing measured it. We built the measurement, pointed it at two writers from different labs, and got a Jaccard of 0.750 over their misses — of everything either one missed, both missed three quarters of it.
Read the note →It could break the code but not test it
The same local model, in the same run, planted 13 bugs that defeated a real test suite and then could not author a single compiling test. Breaking and checking are not one skill, and the seat you put a small model in decides whether it is useful.
Read the note →The compiler was doing the killing
Corral signed a record saying a test suite caught 10 of 13 injected bugs in a file its tests never execute. The suite caught none of them. A mutant that fails to build makes the test command exit non-zero, and that was being scored as a kill — so the number the whole product rests on was crediting the compiler's work to the tests.
Read the note →Four green checks that measured nothing
A week of pointing corral at corral. The security gate that never ran its scanner, the fault highlight showing unmodified code, the critic that reported a clean review while switched off, and the audit gate blind to the one change it exists to catch. Every one looked like a working check from the outside.
Read the note →Prove your tests would catch a real bug, by breaking your code
Your suite passes. That tells you the tests ran, not that they would notice if the code were wrong. Corral answers the question by execution — and hands you the test you were missing.
Read the note →The audit question nobody can answer yet
AI-authored code is entering regulated systems, and no one can show that the model which wrote it did not also grade it. Corral enforces that separation and attests to it — here is exactly what the artifact proves, and what it does not.
Read the note →The fourth participant
corral keeps three models apart so none of them marks its own exam. It never asked who wrote the code — and if that was the same model, the judge is grading its own work. This was the default configuration.
Read the note →Fifty-one minutes
I was going to launch this morning. With the window still open, I finally ran corral against one of my own projects for the first time — and found out it couldn't audit it at all. The post I had written said it supported five languages.
Read the note →The knife was ours
We told Sakana they shipped the capability and left out the containment. Today I found the same shape in my own code: the router that makes 'no model marks its own exam' true could only ever move one seat of three. Named, fixed, and proven cross-vendor.
Read the note →Four words that cost me a day
corral finally proved real bugs in someone else's code — reliably, ten times out of ten. Then it spent an afternoon telling me an audit had failed without telling me why. Plus an honest accounting of how far 'audit a whole repository' actually is: on a 236-file project, corral can currently see nine.
Read the note →A judge may not certify herself. So I made corral certify corral.
The whole thesis is that no one can judge their own work. So I turned the tool on itself and asked it to audit its own kill-rate scorer. It couldn't — for six reasons, each hiding behind the last, all wearing the same mask. Here's the debugging, the six fixes, and the moment it wrote a test for its own code.
Read the note →You can delegate the labor. You can't delegate the accountability.
AI will write your tests now — by the thousand, all of them green. That feels like relief. It should feel like a question: if nobody read them and nobody cares what they do, what is actually deciding your software is fit to ship? A short answer, and a command you can run against your own code in a few minutes.
Read the note →The code is the code
We pointed corral at a real, respected, well-tested project. Its suite killed all eight planted bugs — and then our own critic flagged four of its tests as worthless. The tempting move was to hide the criticism and publish the flattering number. We didn't, and chasing down why taught us the one rule a tool built on trust cannot break: it may only assert what execution proves — including, especially, when the thing making the claim is itself an AI.
Read the note →The critic was never the point
We opened by telling you your tests suck, then spent a note learning to keep that voice honest. Both were about the critic. But the critic was only the door. The point is friendlier and, in the agentic era, urgent: an AI will write you a thousand green tests in an afternoon, and the number of passing tests has never been what makes software stronger — the number that verifiably test something is. corral makes your suite stronger from both ends: it hands you the killing test you were missing, and it proves, by execution, which of your thousand green tests have decayed into no-ops you can delete. And it's built so everyone's tests get stronger from a shared corpus of findings that are verified and signed — knowledge you're actually allowed to trust.
Read the note →Good baking means always mind the D'oh
A builder is bounded by what a model can write. A certifier is bounded by what its sandbox can actually run. Those are different constraints — and confusing them meant we spent a while certifying Go over and over while telling a five-language story. Here's the mistake, the sharper mistake hiding under it, and why the whole thing tastes better for the kneading.
Read the note →We ran our own gate as a stranger (and it bit us)
Nemo iudex in causa sua — no one may judge their own cause — is the whole thesis of the tool. So we pointed corral certify --local at a Python file as a first-time user, not as its authors. It caught the flaw it was built to catch. It also failed its own onboarding in a way only a stranger would ever hit: the absolute path our own README tells you to type. Here's the real run, the Frampton Flaw it found, and the two bugs we found in ourselves fixing it.
Read the note →Nemo iudex in causa sua
No one may be judge in their own cause. It's a maxim of natural justice — and it's the whole design. Most ways of checking AI-written code quietly violate it: the thing that did the work reports whether the work is good. Corral is built the other way, three enforced constraints deep — and here's the objection a good CISO raises, and the answer.
Read the note →Nobody Fails a Test They Never Took
We borrowed one idea from Fugu: route each task to the model that's measured-best at it. Then we built the measuring, and learned that measuring fairly is the entire problem. Three times we penalized a model for the wrong thing — a task it never ran, a success that wasn't its doing, a verdict a human hadn't blessed — and what a fair fitness signal actually takes.
Read the note →Terribly sorry to trouble you… but your tests, um, well. They suck, really
Roger Ebert collected his most withering pans into a book called Your Movie Sucks; John Cleese built a career on the flaw everyone can see and no one can bring themselves to name. I built that critic for your test suite — it mutates your code, runs your own tests against the bugs, and if the tests don't scream it says so (terribly sorry to trouble you), with a signed record and a decorrelated second opinion. Three real verdicts follow: a two-star, a needs-review, and a genuine zero-star.
Read the note →Killing the part I was proudest of
A pivot you don't commit to is just a mood. Three days ago I wrote that corral was an accountability engine, not a builder. Then a friend reviewed the code and emailed me the bottom line: an interesting mashup of the LLM 'harness' and 'router' worlds, with a couple of fun/novel features. He was right — and the couple of novel features were the only part with no incumbent. So we're deleting the builder. Here's why the demo everyone loved had to go, and what corral actually is now.
Read the note →Is any of this worth it?
I lay awake asking whether a very cool thing was a useful thing. By morning the answer was no — not the way we'd been aiming it. We can't win as a builder. But the engine we built to contain agents and run their checks isn't a builder's engine at all. It's an accountability engine. Here's the pivot, and the working proof it produced.
Read the note →What we took from Fugu — and why
We dinged Fugu for shipping the capability without the containment. Fair's fair: here's what it got right, the idea we took from it, and the one honest line about how we took it — the insight, not the mechanism.
Read the note →Fugu, and the knife it leaves unwashed
Sakana's Fugu makes the case that orchestrating frontier models is its own scaling axis. It also leaves out the one thing its own name is about — containing the poison. Field notes on where we agree, and where we diverge.
Read the note →We caught our own gate cheating
The whole pitch here is a deterministic gate you don't have to trust. Then a pre-launch audit found the gate trusting the one thing it never should: the worker's word. Here's the bug, the failing test, and the fix — while nobody's watching.
Read the note →Why we built a brain, not a bot
Agentic coding got real in 2026 — and stayed a black box. We wanted the opposite: a herd you can contain, certify, and query. This is the founding why, and why the whole thing sits on one queryable ledger.
Read the note →