Cheaper classification — implementation plan

Make two local classifier transport changes, preserve safety and existing contracts, then run bounded verification. Implemented in f09db4d1; unmerged, undeployed, PR CI pending.

Design and approval boundary are binding. Author @write Astra high; fresh plan reviews @review Astra high and @review-deletions Grok xhigh; gates @gates Sol low. This lane replaces Fable because kwiss reserves Fable credits. No agents launched during authorship.

Review disposition, 2026-09-10: independent fresh same-family Astra + other-family Grok replacing Fable returned. Orchestrator accepted gate/DSN/retry fixes; rejected global-Gemini routing and slug-only identity false positives. CLASSIFY-only settings and exact response-model/display-name checks stand. Retained scored responses establish GLM completion identity as z-ai/glm-5.3-flash / Wafer, not the dated endpoint metadata ID. Implementation results follow below. The gate waiver below is lane-specific.

Exploration estimate: approximately 16–20 existing implementation/test/config files plus these docs; no new package. Main owners: mail-intelligence, knowledge, worker-mail, worker-ingestion-knowledge. Public context and usage changes include their consumers; avoid widening into a provider or billing framework.

1. Mail HTTP cutover

In packages/mail-intelligence/src/classify-thin.ts, migrate ThinClassifierCtx from Anthropic to an explicit OpenRouter key and structured usage logger. Keep classifyMailThin’s result contract. Implement the spec’s fixed GLM request, deadline, safe errors, completion checks and accounting locally. Remove the dead SDK adapter; preserve prompt/payload/schema/fallback.

Migrate all three callers: apps/worker-mail/src/processors/mail-intelligence-realtime-processor.ts, packages/mail-intelligence/scripts/eval-mail-classify.ts, and eval-mail-exclusion.ts alongside it. Review packages/mail-intelligence/src/index.ts’s wildcard public context export. Keep resolver/gate Anthropic clients and exclusion --model resolver-only semantics. Replace classifier cost counting, not unrelated stage counting. Keep classificationIdFor(..., "sonnet") and CAS stage unchanged.

Gate: update test/classify-thin.test.ts and worker-mail’s processor tests. Exercise real fetch parsing, not only callOverride: valid waiting/private outcomes, malformed/truncated fallback, transport rejection and no private corpus admission. Delete assertions pinning obsolete log wording; retain content-leak detection.

2. Document CLASSIFY and truthful usage

In packages/knowledge/src/pipeline/llm-client.ts, update DEFAULT_MODELS, modelFor, callOpenRouter, MODEL_PRICING, recordStageUsage and snapshots. CLASSIFY resolves only to openrouter:google/gemini-2.5-flash-lite with the spec’s settings/pin, conditional on args.stage === 'classify'; other stages keep their model, reasoning and transport behavior.

Carry returned cost and safe model/provider provenance; otherwise report estimates or unknown counts. Keep numeric known-cost subtotals with explicit completeness/provenance, so callers cannot present partial totals as actual complete spend. Update scripts/eval-ingestion.ts’s usage reporting and any affected snapshot consumers/exports, not DB schemas. Migrate its duplicate resolveClassifyModel, Gemini pin and deterministic interceptor together so the existing eval remains valid.

In src/pipeline/classify.ts, preserve validator and heuristic semantics, remove filename/raw-output/error-text logging, and place configuration preflight outside the fallback catch. Gate in test/pipeline/{classify,llm-client-policy,llm-batch}.test.ts: embedded errors, 429/5xx, body-inclusive timeout, incomplete completion, wrong identity, forbidden provider routing, unknown cost ≠ zero, actual cost precedence, and non-CLASSIFY settings unchanged. Preserve batch regressions on facts rather than deleting CLASSIFY batch guards wholesale. No nested retries.

3. Startup and deployment configuration

Require OPENROUTER_API_KEY before consumers start: apps/worker-mail/src/env.ts/parseWorkerMailEnv and apps/worker-ingestion-knowledge/src/index.ts/main. Reuse a small exported knowledge configuration check for startup/direct ingestion; it is not a provider registry. Test missing-key and unapproved-classify override failures without making requests.

Document OPENROUTER_API_KEY and KNOWLEDGE_CLASSIFY_MODEL=openrouter:google/gemini-2.5-flash-lite in existing package/runtime configuration docs. Preprod’s owner is deploy/preprod/install.sh/build_preprod_env: it copies source environment, then systemd deploy/preprod/systemd/north-os-preprod@.service reads the stable .env.local. Update deploy/preprod/env.preprod.example; preserve key/config wiring, never print secrets. AWS uses scripts/sync-runtime-env.ts and Helm deploy/aws/helm/northos/templates/deployment.yaml’s runtime-secret envFrom. Existing secret ownership needs no new deploy framework. Operator provisions key/config and rolls workers only on kwiss’s deployment decision; no live preprod writes in this lane.

4. Bounded implementation-path quality check

Keep golden labels/prompts unchanged. Mail gates remain chip ≥80%, action recall ≥90%, zero field failures/fallbacks. Exclusion retains internal-sensitive/personal ≥98%, matter recall ≥60%, zero injection leaks. Extend knowledge scripts/eval-ingestion.ts/runClassify to report every exact-scope mismatch, alongside existing type ≥10/12, exclusion 12/12 and no-fallback gates. Accept the reviewed spreadsheet matter → needs_review mismatch; any other scope mismatch needs explicit disposition, and broader/private exposure blocks. No 24/24 perfection requirement or repeated tuning rounds.

Use the real changed classifiers on synthetic fixtures. Update deterministic HTTP interception for Gemini rather than sending incumbent requests. Run exclusion composition offline with controlled gate/resolver results and the actual GLM adapter; the live exclusion CLI still calls Anthropic and is not authorized. Check its migrated caller offline.

Any new paid verification must reserve before dispatch against the same USD10 aggregate meter, starting retained USD1.392817768. Retain failed-attempt reservations and unknown charges; no refunds, incumbent calls, alternate hosts or semantic retries. Mail evaluator wrapper: one reserved attempt per case; disable unmetered retries, leaving production queue retries unchanged. Current documentation work spends nothing. One bounded new-route golden pass is sufficient; unresolved safety failures stop enablement, not trigger an endless benchmark.

5. Gates and handoff

Offline: bun run test from packages/mail-intelligence, packages/knowledge, apps/worker-mail and apps/worker-ingestion-knowledge. Root: bun run lint, bun run typecheck. Lane waiver: do not execute aggregate eval:gate:cheap: it includes unrelated live Anthropic router and DB/Voyage work; retrieval, desktop routing and memory are unchanged, and only GLM/Gemini inference is authorized. Do not change router/cheap-gate source or models to run it. Affected-package tests and direct classifier verification remain required.

If DB suites run, DATABASE_URL, APP_DATABASE_URL and OWNER_DATABASE_URL must all target the same fresh migrated/provisioned isolated clone, never shared/preprod. Invoke scripts directly with Bun, not env-overwriting dotenv wrappers; no new DB harness.

After reservation wiring, bounded paid mail command: from mail-intelligence, EVAL_MAIL=1 bun scripts/eval-mail-classify.ts --runs 1 --concurrency 1. Paid document smoke: throwaway direct production classifyDocument calls over the existing 12 synthetic goldens, applying §4 gates; no DB, no permanent classifier-only CLI/framework. Preserve all outcomes and cost qualifications; do not rerun historical scratch evaluators.

Independent reviews audit additions and deletions to convergence. Refresh existing docs/changelog and remove throwaway verification artifacts after proof. Use bun docs/superpowers/status.ts set cheaper-classification in-progress only when implementation starts; implemented only after gates. Documentation checks: bun docs/superpowers/build-site.ts --dry and bun test docs/superpowers/convert-docs-to-html.test.ts, plus source parsing and pair/index links. No builds/dev servers. Report exclusions and accepted miss; orchestrator owns git, kwiss owns merge/deploy. Rollback is the previous deployment/config, never a retained adapter.

Implementation evidence — 2026-09-10

Product commit f09db4d1 covers 23 product files. Independent Astra + Grok code reviews converged clean after full then targeted reviews; Grok replaced unavailable Fable. Timeout scope and alias identity were fixed; global-reasoning/provider-slug plan findings were rejected as false positives.

Independent offline runs across all four suites: 784 passed, 22 skipped; five knowledge DB files excluded. A subsequent temporal regression was added; final targeted knowledge run: 52 passed (policy 12, classify 23, batch 17), not a rerun of all 785. Root lint/typecheck passed, 64/64 tasks, after local Fumadocs postinstall generated missing ignored .source.

Actual ingestion startup rejected missing key and invalid route before context creation: zero network/server attempts. Approved-route boundary control reached stub context once. The design’s retained implementation JSON records the single approved paid smoke; no further paid runs. Aggregate eval:gate:cheap remains waived under §5.

No real DB tests locally; isolated-DB coverage deferred to PR CI. No UI verification, application build, dev server or deployment. Local gates/reviews satisfy IMPLEMENTED status only; unmerged, undeployed, PR CI pending.