Star
Search corralai.dev

Limitations & roadmap

Corralai is v1.0.0-rc.29 — solo-maintained and moving fast. This page is the honest version: what it does today, what it can’t do yet, and what’s next.

What it is today

  • Certify by execution, end to end. corral certify --local (one file) and corral certify --repo (a whole repository, bounded by --top, --path or --diff-base) run the adversarial pool: mutant-generator → your suite in the jail → test-writer → decorrelated test-critic. The kill rate is your suite’s exit code against each planted fault; a proven gap is a survivor the test-writer killed with a test that also passes on your real code, both run. The test-writer does corrective retries, feeding the compiler’s own error back on a non-compiling attempt.
  • Seven languages — Go, Python, Ruby, JavaScript, TypeScript, PHP and Rust — with the runner read from your test command or its package script. Evidence is uneven and stated plainly: Go and Python are exercised hardest; the others rest on a single third-party repository each so far (Rust on Warp’s crates/fuzzy_match, PHP on webmozart/assert). The evidence table is in the README, and what each runner supports is on language support.
  • Any seat, any provider, or a subscription. corral has no default models. Each seat is an API model you name or a coding-agent CLI (claude-code, codex, or one you define) running on its own login — or a local model through Ollama, with no key at all.
  • A merge gate. certify --repo --min-kill-rate / --max-proven-missed turn a scan into an exit code any CI can gate on; the GitHub Action wraps it.
  • The record. Every certify --repo and corral review run writes a hash-linked entry, signed when a certify key is configured, to a ledger directory (in CI, the corral/ledger branch); --attest --transparency logs the signed statement to Sigstore Rekor. corral verify checks three legs — DSSE signature, warehouse rows-hash against the statement, Rekor inclusion — and has run against live infrastructure: Rekor index 2667058567, from a real flask audit.
  • Analysis over the record. --push appends verdicts to a DuckDB or MotherDuck warehouse you own (corral_scans, corral_audits, corral_mutants, corral_model_calls, corral_events; a review push adds corral_reviews, corral_findings and corral_adjudications; a corral_seal view cross-checks the statement hash), the DuckDB page queries the public record in your browser, and corral models rank ranks the models that have sat in each seat by recorded evidence.
  • Reuse and replay. Recorded mutant sets (--record-mutants / --mutants), priors from earlier runs (--prior), and goal, selection and verdict caches keyed on content, so a second scan of an unchanged tree is close to free — and says how old the reused verdicts are.
  • Coverage-evidence candidacy. A file coverage evidence measured at zero covering tests is excluded under an honest name (uncovered — no test executes this file, imported at load time, no executable code) rather than graded against nothing; --preflight reports which source files the suite never executes.
  • time:/cost: lines on every run, and --max-tokens as a hard token cap across every audit seat (corral review has none).

What it does not claim

See also Trust & security and SECURITY.md.

  • It does not adjudicate a survivor as a bug. A survivor is a planted mutant your suite didn’t catch — corral discloses it unadjudicated. Whether it’s a real untested edge or an equivalent mutant (a change nothing can observe, and so no test can catch) is the human’s call, not the tool’s. corral only calls a survivor a proven gap when its test-writer authors a compiling test that actually kills it. And it does not promise to prove one. On real third-party code, some runs manage it and some don’t. Zero proven gaps means “nothing was proven in this run”, never “your tests are fine.”
  • A kill rate is a sample, not a constant. Mutants are generated fresh each run, so an unchanged file scores differently run to run; --mutants pins the set when you need a comparison. Proven-gap discovery can vary too: a recorded pair remains evidence, but a fresh run may not find it. Writer success and flaky execution can vary even when the mutant set is fixed.
  • A confidence interval is not confidence in software correctness. The Wilson calculation assumes independent observations from a representative sample. Goal-directed generated edits do not establish those assumptions. Its exam-size gate rejects thin evidence; it does not turn the exam into an exhaustive audit. See the measurement limits.
  • Individually proven tests are not a merge-ready bundle. Review their assertions and run them in your full suite. On unreleased main after rc.24, per-survivor runs with multiple proven tests also check the merged file once. authored_together: fail warns without changing the individual proven count or certification status; an absent field means not checked. Passing once does not establish freedom from flakiness.
  • A ledger is not immutable or automatically complete. Entries can be unsigned when no certify key is configured. The owner can rewrite unsigned hashes or delete files; a missing tail requires an externally held expected head to detect. A trusted signature authenticates bytes, not the truth of an audit’s claims.
  • It does not make a model immune to instructions in the code it reads. The control is that no model’s output decides a verdict: the kill rate and a proven gap are execution, and the critic is advisory and never gates.
  • It has not been battle-tested at scale. The properties are proven by design review and adversarial tests, not (yet) by a hostile production adversary.

Current limits

Named, so they are not discovered one paid run at a time:

  • The sandbox is offline, and sees /usr, your checkout and its dependency dirs — not your home directory. Anything your suite fetches from the network fails inside it, and a toolchain installed under $HOME (pip install --user, pyenv, nvm, a stock rustup) is invisible; install it under /usr, put a Python virtualenv inside the project, or use --substrate workspace (below). See Getting started.
  • On macOS the sandbox is weaker. sandbox-exec runs as your user, and a jailed command can read the start-up environment of your other processes through the kernel (measured on macOS 27, 2026-10-13; no Seatbelt rule tried blocked it). Do not audit there with API keys exported in other shells, or use --jail container. See SECURITY.md.
  • --substrate workspace mutates your real checkout. It is opt-in, for a place that is already the isolation boundary (a CI runner, a scratch clone). Restores run on a failing command, a timeout, a panic and an interrupt; a process killed outright (kill -9, out of memory, a reclaimed runner) can leave a mutant behind, and nothing detects it. Commit or stash first.
  • Your test command must collect the test corral writes. corral writes its authored test as a new file beside your paired test. A command pinned to one file never collects it, and a project that confines discovery (pytest’s testpaths, jest’s roots) may not either. corral checks this by planting invalid source at that path; when your command does not react, the file is reported [TEST UNSOUND] and its proven count is withheld rather than printed as zero. Scope the command to a directory, not a file.
  • One language per explicit test command on certify --repo: a command after -- grades every file, so it is refused when the scan spans more than one language. Omit it, or narrow the scan with --path.
  • Rust: a source whose tests are an inline #[cfg(test)] mod tests { … } block cannot be audited yet (corral would have to compose its test into the source file itself); it must declare its test module in a separate file. Rust also has no fail-fast flag verified, no salvage of a partly failing authored test and no coverage pre-flight.
  • Coverage-guided test selection — grading each file against only the tests that execute it — is Python-only. Every other language grades against the test command you give.
  • The tests×mutants matrix (--matrix) is Go and Python only.
  • The critic must judge every test only in Go, Python, PHP and Rust, where corral can list a file’s tests statically; in JavaScript, TypeScript and Ruby it judges the tests it chooses to name.
  • Repository scans pair by convention and coverage, and some projects defeat both. A project that names tests after behavior rather than after source files pairs poorly: expressjs/express pairs zero files by convention, and 6 with a --tests map (JSON, source → test file) — both pinned in CI (testdata/foreign-sweep-expected.tsv). A 0 candidates report on such a project is a pairing limitation, not a verdict on your tests: hand it --tests, or audit files directly with certify --local --repo-dir . --code <path> --test <path>.
  • The certified/needs-review bar is fixed at a 0.8 kill rate. A per-repo gate is what --min-kill-rate and --max-proven-missed are for; prefer --max-proven-missed, since a kill rate moves between runs on unchanged code.
  • Agent seats: a seat on the text wire (antigravity, or a definition that declares no wire) reports no token usage, so it shows not metered and is refused under --max-tokens. The agent CLI inherits corral’s environment, so an exported vendor API key may be billed instead of the subscription. codex’s read-only sandbox stops writes, not reads: it can still read the host. A subscription limit has not yet been observed on either CLI, so how one is reported is unverified. See agent seats.
  • Decorrelation is enforced only as critic ≠ writer. Two models from one vendor satisfy it; corral warns when every graded seat is one vendor, and does not refuse, so a single-vendor operator is not stranded.
  • There is no dollar cap. --max-tokens caps tokens; prices differ by contract, so the cost line reports tokens and calls, never dollars.
  • No pentester role in the audit. The pool runs mutant-generator, test-writer and test-critic; it plants faults, it does not hunt for vulnerabilities.

The optional daemon

These apply only to the brain (corral-wrangler), which is frozen and which no audit uses:

  • One active adversarial-pool run at a time, per brain. start_adversarial_run refuses a second run while one is active; certify --local and --repo have no such limit.
  • Its repo gate posts commit statuses on GitHub only. The GitLab and Gitea providers are complete but return “unsupported” for listing open pull requests and setting a commit status. A certify --repo exit code gates any CI (see internal forges).
  • It assumes a single trust domain, and does not defend against a malicious insider holding a valid key.

Where it’s heading

From the roadmap (directional, not committed):

  • Wider pairing by default. Coverage-derived pairing ships, but needs --preflight to have run; a repository audited without coverage still falls back to filenames.
  • The runner table’s remaining phases: static test lists for JavaScript, TypeScript and Ruby, Rust fail-fast and salvage verified against real runs, cargo llvm-cov for Rust coverage, and per-test coverage selection beyond Python.
  • Cost governance. Dollar estimates before a run, and caps per role or model.
  • Many audits at once — scheduled, isolated, fair.
  • A review cockpit docked beside the diff.

The through-line: every capability is judged by a party that did not produce the thing, checked by execution where possible, and recorded with its evidence and limits. The repository owner controls the ledger; a third party verifies it against a trusted key and any external anchors, not against an ownership claim.

Adoption and comparative evidence still to gather

Maintainer-run audits demonstrate individual executions, not independent adoption or product-market fit. This documentation does not establish a repeat-use study by outside maintainers or a head-to-head advantage over other test-generation tools. The related-work discussion names prior approaches; citations are not benchmarks.

An outside trial should retain unsuccessful attempts as well as useful results: setup failures, time to first useful test, API spend and runner time, proposed tests reviewed, tests accepted as useful and nonredundant, rejection reasons, review effort, and voluntary repeat use. The useful outcome is cost and elapsed time per accepted test, not just mutant count. No results for that trial are claimed here.

Want to shape it? Issues and verified-harness PRs are welcome on GitHub.