Running it

Dev mode vs auth-on

With CORRALAI_OIDC_ISSUER unset, the brain runs with auth disabled — any caller is trusted. This is for local development only. Setting CORRALAI_OIDC_ISSUER to any OIDC provider (Keycloak, Auth0, Okta, Dex, Authentik, or others) turns auth on: callers must present a valid token for that issuer and the configured CORRALAI_OIDC_AUDIENCE. Dev mode and auth-on run the exact same code paths, so “works on my machine” and “works with auth” don’t drift apart. See the full env reference on corral’s CLI page — every variable there is generated straight from the binary’s own source comment, not hand-copied.

The demo compose profiles

corral certify --local (see Getting started and Your first audit, in detail) is the fastest way to see the audit itself — no daemon required. deploy/demo/ is the separate, optional compose stack for the coordination substrate underneath a running brain: it brings up the brain plus a small herd of agents against a bundled local model, no API keys required. make demo is the base coordinated herd, make demo-clobber runs the same herd with claims turned off (a demo of what NOT coordinating looks like), make demo-observe adds a read-only observer console, and make demo-models runs a two-model comparison so model_comparison has real A-vs-B data to report on.

The scorecard + critic precision (brain-path only)

corral scorecard shows which model actually catches bugs, execution-proven (recall from mutants a model’s own generated tests killed — never a self-report) and, in a C-PREC column, how often that model’s test-critic findings turn out to be real once a human adjudicates them. Both columns read the running brain’s DuckDB stores over its HTTP API (GET /api/bugcatch) — set CORRAL_BRAIN (and a token via corral secret) to point at one; without it, corral scorecard falls back to opening the local CORRALAI_BUGCATCH_DB file directly, which shows recall/precision but not the C-PREC column (that join needs CORRALAI_CRITICSCORE_DB, and the offline path never has both single-process files open at once).

The critic’s own findings get flagged automatically the same way the rest of the audit works — by execution, not opinion — but the human gate has final say: corral criticscore list shows findings still awaiting adjudication, corral criticscore show <id> prints one in full (model, target test, evidence), and corral criticscore confirm <id> / refute <id> record a human verdict that permanently overrides the auto-adjudication — a later automated pass can never claw a human verdict back. All four subcommands require a brain (CORRAL_BRAIN); there’s no offline mode, because confirm/refute are admin-gated MCP tools that need the caller’s authenticated identity for the audit trail.

Honesty caveat: the critic-precision metric and its human gate are brain-path features only, like the rest of the bug-catching scorecard. corral certify --local still shows the critic’s auto-adjudicated verdict on the run’s own tape (nothing is hidden from you locally), but it persists nothing to a scorecard — there’s no server-side store to write to without a brain running.

The tests×mutants matrix (--matrix, opt-in, brain-path only)

corral certify --local --matrix opts a run into a second, more expensive pass: after the primary mutant pass converges, it re-scores every one of the dev suite’s own tests, alone, against the same mutants — a per-test adequacy number instead of one suite-wide kill-rate. The verdict prints a matrix: summary (tests scored, delete-candidate count) and each delete-candidate selector, and the run’s --record tape carries a pool_matrix beat with the same data.

The cost is real and it’s why this is opt-in. The matrix is T tests × M mutants extra jail runs, on top of the primary pass — a suite with 40 tests and 40 mutants is up to 1,600 additional sandboxed executions. Leave it off for routine runs; turn it on when you actually want the per-test signal (e.g. before a cleanup pass on a suite an agentic dev grew fast).

The safe-to-delete list is a signal, not a verdict. A delete-candidate is a test that was scored (compiled and ran clean against the compliant code) but killed zero of the run’s planted mutants. Every delete-candidate line carries the same honest caveat, verbatim: “Relative to this mutant set; a test may still guard behavior no mutant probed.” Zero kills against this run’s mutants is not proof the test is vacuous — it means this run’s mutant generator never happened to seed a violation that test would catch. Review before deleting; don’t script deletions off the list unattended.

Persistence is brain-path only. A --matrix-opted run’s per-test rows land in the running brain’s internal/matrixstore DuckDB store (CORRALAI_MATRIX_DB) — the same store the hosted adversarial pool’s start_adversarial_run writes to when its own matrix param is set. corral matrix list [--json] reads it back over the brain’s HTTP API (GET /api/matrix, mirroring /api/bugcatch); like criticscore, there is no offline fallback — set CORRAL_BRAIN (and a token via corral secret). A --local --matrix run with no CORRAL_BRAIN configured prints its own tape/verdict summary same as always, but that run’s rows are never persisted anywhere corral matrix list can see them.

Language support: go and python only today, the same two languages the underlying test-enumeration/single-test-selector plumbing (ListTestsCmd/ SingleTestCmd) currently implements — an unsupported language silently skips the matrix phase (the primary audit still runs and gates normally).

Python single-file caveat: in single-file --test mode (no repo dir), enumeration runs pytest --collect-only against the operator-supplied --test path itself, not the synthetic path the dev test is actually written to in the jail — so on Python, the --test filename must follow pytest’s discovery convention (matching the synthetic test_<code>.py path) or collection finds nothing and the matrix is silently skipped (fail-soft, no crash). Repo-dir mode and go are unaffected.

Verdict reuse — the cache, and what it will and will not do

certify --repo --record consults a content-addressed cache before auditing a file, and reuses a verdict when every input that could change it is unchanged: the file’s contents, its package, its goal, the test surface, the substrate, the resolved model set, the verdict-affecting flags, and a hand-bumped engine generation. Change any one and the file is re-audited.

Reuse is always disclosed. The report names how many verdicts were reused and the age of the oldest one, because a scan that reports reused work without saying how old it is presents stale measurement as current — the exact self-flattering record this tool exists to prevent. The ledger marks each row with cache_hit, so an aggregate that grades models can exclude reused rows rather than counting one measurement once per scan forever.

It fails closed. An unreadable row, verdict JSON that will not parse, a missing store or an empty owner all resolve to a miss. A miss costs money; a wrong hit signs a claim about content that was never measured.

Two things decide what it is actually worth to you, and both are easy to be surprised by:

• Any change to any test file invalidates every file’s verdict on the whole-suite path. That is correct — the grading surface really did change for every file — but it means the hit rate is lower than the word “cache” suggests.
• Goals are derived per run unless you pin them. The goal is part of the key, and an LLM-derived goal is not byte-identical between runs, so without --goals the cache is largely inert. If you want reuse, check your goals into a file and pass --goals.

Known gaps, documented here because they are wrong-hit paths rather than design: the file-scoped path (coverage-guided selection, or a test command naming specific files) does not key on the wider test surface, so weakening a shared tests/conftest.py leaves the key unmoved; a repo-root conftest.py with tests under tests/ is not covered; and argv tokens given as absolute paths or symlinked directories do not disqualify a file-scoped scan.

What a run prints while it works

A run echoes the pool’s own beats to stderr as they happen — a seat starting, mutants landing, your suite being scored, the verdict signed:

· auditing kirby/characters/xp_service.py
· mutant-generator/3 started (seat 3)
· mutant-generator/3 produced 5 mutant(s)
· scored your suite: killed 22 of 39
· verdict signed

These are the same events --record captures as a replayable tape. They go to stderr so nothing parsing stdout — the verdict, --out — is affected, and --quiet turns them off.

A certify --local run also ends with what it consumed:

model spend:   18432 in / 5120 out token(s) over 41 model call(s)

Tokens, not dollars: prices change and differ by contract, so multiply by your own rate. The call count is printed alongside because a provider that reports no usage leaves the token counts at zero, and “0 tokens over 41 calls” is a different fact from “0 over 0”.

Dependency dirs — bound read-only, never copied (--repo-dir)

corral certify --local --repo-dir <path> seeds the whole cloned tree into the jail workspace so a real multi-file project’s own tests (which import the package) resolve. Dependency dirs are the usual bulk of that checkout and are irrelevant to the mutant/text seed, so instead of copying them in, corral auto-detects and bind-mounts them read-only: node_modules, vendor, .venv, venv, and .bundle. This keeps them off the workspace’s 64 MiB text cap and off the mutant/test-writer’s context.

Dependencies must already be present — vendored or installed, the same way CI expects a checkout to arrive. Corral binds what’s already there; it never installs anything on your behalf.

• --bind-dir <path> (repeatable, repo-relative) binds additional dirs the same way, for dependency trees the auto-detected set doesn’t cover.
• --no-bind-deps opts back into the pre-bind behavior: every dep dir is copied into the workspace seed instead, subject to the same size cap.
• When anything is bound, the run prints deps: bound N dir(s) read-only (<names>) — not copied into the jail seed.

The honest backend caveat. Binding a dep dir requires the backend to relocate it into the jail workspace — only bwrap (Linux default, via --ro-bind Host Target) and --jail container (via -v Host:Target:ro) can do that. bwrap binds cleanly with no size limit and no copy. --jail container is more restricted: containers remap the host uid, so a container process can’t read a dep dir it doesn’t have world-read permission on — --jail container binds only world-readable dep dirs; anything more restrictively permissioned is copied into the seed instead (and so is subject to the workspace size cap) — a loud fallback, not a silent one.

macOS sandbox-exec cannot bind dep dirs at all. sandbox-exec only grants read access at a dir’s original host path — it has no primitive to relocate a dir into the jail workspace, which is where the test toolchain (cwd = the workspace) needs to find it. So on macOS (and on any other/unknown backend), every dep dir is copied into the workspace seed, the same as --no-bind-deps, and is subject to the 64 MiB workspace size cap.

The verify gate needs a sandbox (Ubuntu 24.04+)

The brain certifies a gated task by running its verify command itself — in a jail, against its own working copy, reading the real exit code — rather than trusting what a worker reports (see Trust & security). That jail is bubblewrap by default, and bubblewrap needs unprivileged user namespaces.

On Ubuntu 24.04+, unprivileged user namespaces are restricted by default (kernel.apparmor_restrict_unprivileged_userns=1). If the brain can’t get a sandbox it does not crash and does not silently trust the worker without saying so — it logs, loudly, and falls back to the recorded-execution lookup:

verify-gate: NO isolation backend (bwrap ... Permission denied);
gated completion falls back to worker-reported executions — set CORRALAI_GATE_EXEC_BACKEND

To get the independent gate, install bubblewrap and grant it (and it alone) the userns permission with a small AppArmor profile — this keeps the rest of the host hardened:

sudo apt install -y bubblewrap
sudo tee /etc/apparmor.d/bwrap >/dev/null <<'PROF'
abi <abi/4.0>,
include <tunables/global>
profile bwrap /usr/bin/bwrap flags=(unconfined) {
  userns,
  include if exists <local/bwrap>
}
PROF
sudo apparmor_parser -r /etc/apparmor.d/bwrap

Restart the brain and confirm the log now reads verify-gate: independent verification enabled (backend bwrap). The profile in /etc/apparmor.d/ reloads automatically on boot.

Alternatives to bubblewrap: set CORRALAI_GATE_EXEC_BACKEND=container to run the verify command in a container instead (needs a runtime and an image carrying the project’s toolchain), or — only on a host that is already a disposable sandbox — CORRALAI_GATE_EXEC_BACKEND=none with CORRALAI_GATE_EXEC_UNSAFE_HOST=1. Lifting the restriction globally (kernel.apparmor_restrict_unprivileged_userns=0) also works but weakens every process on the host, so prefer the per-binary profile.

Deploying a brain behind a tunnel

The brain listens on CORRALAI_ADDR (default 127.0.0.1:9019) and expects to be reached through a reverse tunnel or proxy that terminates TLS and forwards to that local address — the general pattern is: run corral bound to loopback, run a tunnel client (any provider) pointed at that same loopback address, and set CORRALAI_ALLOWED_HOSTS to the public hostname(s) the tunnel exposes so the brain’s Host-header check accepts them. Corralai supports built-in TLS via CORRALAI_TLS_CERT/CORRALAI_TLS_KEY, or CORRALAI_TLS_AUTOCERT_DOMAINS for automatic Let’s Encrypt certificates, as alternatives to a tunnel for a brain with a public IP. This page describes the pattern only — specific tunnel providers, hostnames, and ports are a deployment’s own operational detail, not part of this doc.