Eval gate split — implementation plan

Status: In progress · Author: Claude (NOS-216 orchestrator) · Date: 2026-07-23 · Repo: north-os
Related: Linear NOS-216; design spec 2026-07-23-eval-gate-split-design.html

Superseded 2026-08-26: the agent-quality gate was removed from the deploy path and the ratchet, its weekly runner, and its baseline were deleted.

Order of work

Measure the five cheap batteries. Run each locally, record cost, wall time and today's score. This decides which can run on a hosted runner and which need the corpus, and gives the PR gate a known-green starting point.
Battery report contract. scripts/eval-battery-report.ts — BatteryReport/EvalBaseline types plus pure buildBaseline, compareToBaseline, formatRatchet. Pure and DB-free so it unit-tests without a runner.
--report / --only on eval-agent-quality.ts. Per-case verdicts to JSON (a case passes only if it passed every run) and a case filter for the confirmation re-run. No change to how any case is judged.
scripts/eval-ratchet.ts. Check mode and --record mode, with the confirmation re-run and the refusal to lower the bar without --allow-regression --reason. Scripts: eval:ratchet, eval:baseline:record.
Unit tests. eval-battery-report.test.ts + eval-ratchet.test.ts.
Runner setup. deploy/ci-runner/{install.sh,north-os-ci-runner.service,README.md}. Committed and documented, never a manual folk step.
Workflow rewrite. changes / cheap-gate / corpus-gate / eval-gate plus .github/scripts/eval-changed-batteries.sh.
Deploy gate. deploy/preprod/deploy.sh step 4d (ratchet) and 4e (opt-in trajectory smoke), EVAL_CORPUS_DATABASE_URL in env.preprod.example, README runbook.
Record the baseline. One full eval:baseline:record run; commit the JSON.
Gates, PR, CI green, merge.

Files

Path	Change
`.github/workflows/eval-gate.yml`	rewritten — four jobs (shared edit point: keep additions on their own lines, rebase before final gates)
`.github/scripts/eval-changed-batteries.sh`	new — per-battery path filter
`deploy/ci-runner/install.sh`	new — idempotent runner install
`deploy/ci-runner/north-os-ci-runner.service`	new — systemd user unit template
`deploy/ci-runner/README.md`	new — runbook, security posture, uninstall
`deploy/preprod/deploy.sh`	steps 4d + 4e before the fleet restart
`deploy/preprod/env.preprod.example`	`EVAL_CORPUS_DATABASE_URL`
`deploy/preprod/README.md`	gate documentation
`packages/agent-runtime/scripts/eval-battery-report.ts`	new — report/baseline contract + pure comparison
`packages/agent-runtime/scripts/eval-ratchet.ts`	new — the gate runner
`packages/agent-runtime/scripts/eval-agent-quality.ts`	`--report`, `--only`
`packages/agent-runtime/eval/baselines/agent-quality.json`	new — the committed contract
`package.json` (root)	`eval:gate:cheap` extracted, `eval:gate` composed from it, `eval:ratchet` (shared edit point)

Verification

bun x vitest run scripts/eval-battery-report.test.ts scripts/eval-ratchet.test.ts — the ratchet's decision logic.
bash -n on all three shell files; the filter script exercised against a real polaris-only commit (must select nothing) and against this branch (must select everything).
The five cheap batteries run locally, scores in the PR body.
eval:ratchet run end-to-end against the freshly recorded baseline — must be green, and must be green while the battery's own absolute gate is red.
bun run lint, bun run typecheck, bun --filter=@workspace/agent-runtime test.
The workflow proves itself on its own PR run.

Risks

Runner registration needs repo admin. Minting the token is the one step the orchestrator cannot perform; it is handed to the human with the exact command. Until the runner is online and EVAL_SELF_HOSTED_RUNNER=true, the corpus batteries skip loudly — never silently, never hanging.
Judge flicker reddening a deploy. Mitigated by the confirmation re-run; a case must fail twice.
Deploy time and cost. The ratchet adds ~20 min and ~$15–30 per preprod deploy. Preprod deploys are morning-only and infrequent; EVAL_GATE_SKIP=1 exists for incidents.
Shared edit point. eval-gate.yml and the root package.json are also being edited by the parallel Mail-RAG / learning-loop factory. Rebase on origin/main before the final gates and re-run them.