Outlook automatic recovery — implementation plan

All seven slices implemented; final Astra and Opus finders CLEAN, no new actionable regression; NOT merged or deployed. Historical correction checkpoint 30dd5ad1, with subsequent corrections below. Metadata stays in-progress for lane acceptance pending. CI is not yet available and remains pending; NOT MERGE-READY. Clean reviews confer no merge/deploy authority. The design spec preserves acceptance and incident provenance; the durable verification report records exact evidence and limits.

Latest product/evidence checkpoint: 68452506, reported pushed by the user; 30dd5ad1 is historical only. No deployment or CI result is inferred. Final cleanup adds one existing-format changelog entry for this checkpoint, not product behavior.

Scope and invariant

Challenge → typed queue hint → guarded renewable-grant snapshot → ONE connector-outlook maintenance primitive → provider-proven two-folder coverage → ONE guarded recovery writer → bounded catch-up. Recovery admits authorized or unconfirmed requires_reauth; exact confirmed markers instead permit renewal-only. Pending/error also permit renewal-only active-row PATCH, never CREATE, authorize or catch up; revoked/archived excluded. Missing/dead/interaction-required user grants require action; transient/network/429/5xx/invalid_client app faults do not.

Review prerequisites

1. Historical source/deletion reviews informed the plan, but Grok was subsequently deemed untrusted; Astra substitution was user-directed. Final Astra and Opus finders are CLEAN. Full reviews covered deletions and structural changes; the last finder pass covered only the final renewal predicate. Lane acceptance and CI remain pending.
2. Dependency provenance: kwiss reports #400 merged at 7fa680c3; this branch is based on it. Historical private-branch integration fast-forwarded ONLY fix/outlook-automatic-recovery from f30b280b to 457741e3d024a1bdcea7e519293c56beee6f11f1, then the feat/connector-health head with 8 successful CI checks. Its former OPEN/MERGE-READY state is superseded and is not recovery CI evidence. Read-only source was /home/kwiss/workspace/north-os-connector-health. Public @workspace/db withConnectorStatusTransition is reused without copying or resetting authorized_at. No worker git actions.
3. The slice briefs below retain their implementation instructions and original caps as historical acceptance criteria, not remaining unchecked work. Current contracts and residual limits supersede stale baseline descriptions; no new resource resolver, durable lease/schema or general auth framework.

Bounded implementation slices

All seven slices below are implemented. Original caps counted authored additions plus deletions, including tests, excluding generated output; they remain historical stop limits, not a retrospective size certification. Slice checks mean implementation, not final review/CI approval. “New” paths describe the original brief.

1. Lifecycle and queue adapter — ≤4 files / 360 lines

• packages/worker-chassis/src/mail-subscribe-queue.ts, existing packages/worker-chassis/test/mail-subscribe-queue.test.ts; apps/web/lib/connections/outlook-lifecycle.ts and its existing outlook-lifecycle.test.ts. Owner: producer/queue worker.
• Typed reasons on the existing queue: challenge enqueues without downgrade; removal revokes exact matched row then enqueues repair+delta; missed retains delta. Keep identity/client-state checks, null-user rejection, unknown events and batch isolation. Fresh BullMQ IDs, three attempts/exponential backoff, explicit recreate/catch-up payloads; no outstanding-job coalescing or permanent retained-job IDs.
• Prove duplicate/removal deliveries serialize downstream without losing catch-up; an event after the active job's delta walk schedules later work, even with completed/failed jobs retained. Keep OAuth's explicit 60-day lookback and worker one-day fallback; no 60-day payload on lifecycle/scheduled jobs. Queue outage stays visible for backstop repair.

2. Guarded auth outcomes and credential persistence — ≤10 files / 1,250 lines

• packages/connector-auth/src/tokens/{outlook-recovery.ts,refresh.ts,encrypt.ts}, src/index.ts; packages/connector-auth/test/{outlook-recovery.test.ts,refresh-microsoft.test.ts,refresh-revoke.test.ts,refresh-scope.test.ts}; packages/connectors-core/src/outlook-recovery.ts (browser-safe contract) and packages/connectors-core/package.json (explicit subpath export). Owner: auth worker.
• ONE dedicated recovery writer; import exact #400 helper only on integrated base. Its connector FOR UPDATE precedes callback; after Graph success lock BOTH exact proven subscription rows in deterministic ID order inside that transaction. Verify org/owner/connector/provider IDs, active/unexpired status and captured post-write generations, then connector identity/status/updated_at CAS. Preserve authorized_at; zero rows is stale, not success; same-tx history insertion failure rolls back. Omit errorCode for success; no network I/O in the transaction.
• Use exact Postgres timestamp strings and existing GREATEST(clock_timestamp(), updated_at + interval '1 microsecond') generation convention, never Date round-trips. Reuse PersistTokensTransactionHooks.beforeWrite in persistTokens' transaction: connector lock/generation then credential lock/snapshot fence before write. Examine precise read/decrypt/write metadata seam, including lazy backfill, to reject deleted/replaced credentials even if concurrent OAuth replacement has not yet advanced connector generation. No replay upsert after disconnect; no durable lease/schema.
• Recovery requires a refresh grant even with a usable cached bearer. Guard the actual missing-row read in loadTokens and missing refresh_token with a typed missing-grant outcome; loadTokens reads the user/provider token row including account_id inside the organization's RLS transaction. Malformed/decryption/config errors are not consent loss. Preserve ordinary callers' cache behavior and narrow authorized-only revocation/anchor semantics. Reuse isDeadGrantError; confirmed interaction outside that allow-list stays a dedicated outcome, never broad class/message matching.
• Apply the design's connector-before-token snapshot/absence fence to ALL negative outcomes, including reason-only; route/suppress the legacy pre-persistence downgrade only in typed recovery context.
• Typed one-time forced refresh carries the rejected bearer in memory only. Fast cache, in-lock reread and contention polling reject that bearer but accept a different usable token; scopes checked everywhere. Require Mail.ReadWrite for maintenance; preserve union refresh scopes. One rejection refresh retry per whole maintenance attempt, not per folder. Add real abort/deadline to token endpoint fetch; reuse netdocs.ts owner-checked Lua/deadline pattern, never non-atomic GET/DEL release.
• Define exact closed recovery reason constants once in the browser-safe public contract specified by the design. Guarded writer handles authorized→requires_reauth AND reason-only updates on already-requires_reauth; transients never overwrite confirmed reasons, success clears only exact recovery-owned markers. Regressions: all three cache paths, replacement before connector bump, replay-after-disconnect, missing token row, missing refresh with fresh bearer, microsecond CAS and subscription revocation before final CAS. Preserve #400 revoke/lifecycle/API helper hunks.

3. Single provider-proven maintenance primitive — ≤6 files / 1,150 lines

• packages/connector-outlook/src/{subscriptions.ts,index.ts,errors.ts}, existing packages/connector-outlook/test/subscriptions.test.ts; packages/provider/src/adapters/msgraph/subscriptions.ts and existing packages/provider/test/msgraph-subscriptions.test.ts. Owner: connector/provider worker. No second package or worker orchestrator; implement the primitive in existing subscriptions.ts.
• Shared org/connector serialization covers ensure/create/renew and every exported entry point, including OAuth creation/operator force-recreate. Recovery admission allows CREATE/CAS/catch-up; pending/error renewal-only admission allows guarded PATCH of existing active rows only; revoked/archived excluded. Thread actual AbortSignal/deadlines through provider PATCH/CREATE/DELETE, keeping SDK retry handler/Retry-After; no Promise.race pseudo-cancellation or overlapping retry engine. Abort on lease loss/deadline and reject subsequent writes.
• Exact anchored parser only for demonstrated me/users/<id> well-known Inbox/SentItems forms; org/connector/owner-scoped rows. Unknown/opaque resource stays unresolved, never healthy or guessed-folder deletion/recreation. Duplicate CREATE alone proves nothing without actual provider reconciliation and client-state custody. Active-but-expired/null-expiry coverage is not healthy; valid returned expiry is required. Keep default 69.5-hour lifetime.
• PATCH/CREATE BOTH required folders before healing; partial success never authorizes. Capture exact guarded post-write row generations for slice 2's final locks/CAS. Graph 404 retires the exact old row BEFORE CREATE under recovery admission (active-row unique index); renewal-only never creates. Keep encrypted client state, orphan/no-token expiration, local/provider cleanup and failure isolation.
• Before CREATE, apply the design's known revoked/expired reconciliation and last_renewed_at fence; retain exact provider/local proof and final deterministic row locks.
• Baseline CREATE string-wrapped errors after partial rollback; implementation preserves typed error/cause for bearer rejection and structured throttle/conflict. Keep partial CREATE cleanup and cleanup-failure warnings; cleanup only this attempt's newly created IDs on stale completion. No provider response fields beyond id/resource/expirationDateTime/changeType and no delivery claims.
• Tests first in existing suites: two/one/zero folders, 404 unique-index ordering, unknown resources and duplicate conflict unresolved; rejected bearer on CREATE as well as PATCH; one refresh across folders; actual transport abort; removal between PATCH and local write, and after proof before CAS; pending/error PATCH capability without CREATE/CAS/delta. No new subscriptions-recovery suite duplicating existing coverage.

4. Thin worker convergence adapter — ≤2 files / 650 lines

• apps/worker-mail/src/processors/outlook-subscribe-processor.ts and existing apps/worker-mail/test/processors/outlook-subscribe-processor.test.ts. Owner: worker adapter worker; no new worker maintenance orchestrator/module.
• Replace authorized-only/any-active shortcuts with identity-scoped admission and delegation to the package primitive. Jobs are hints; use its guarded snapshot/proof/outcome, not independently duplicated locks, refresh or folder orchestration. Pending/error renewal-only hints never trigger CREATE, authorization or catch-up.
• Preserve operator force-recreate semantics under shared serialization and protected pending enrollment. OAuth supplies 60-day firstConnectLookbackDays; missed/catch-up retains one-day fallback. After verified coverage/current recovery admission, removal, 404/replacement or missing coverage forces bounded delta even when the original hint omitted its flag.
• Tests first: provider failure cannot heal; both-folder proof permits guarded CAS then delta; stale/zero-row CAS prevents success; concurrent reconnect/disconnect wins. Exercise late catch-up event with separate fresh-ID work. Dead/missing grant stays disconnected with typed reason; transient/app failure preserves state and confirmed reason.

5. Periodic discovery and renewal cutover — ≤5 files / 800 lines

• apps/worker-mail/src/loops/{outlook-scheduled-resync.ts,outlook-subscription-renewal.ts,outlook-scheduled-resync.test.ts}; existing apps/worker-mail/test/loops/{outlook-scheduled-resync.test.ts,outlook-subscription-renewal.test.ts}. Owner: scheduler worker.
• EXPAND authorized queries to authorized+requires_reauth and missing/partial/expired coverage. Preserve authorized zero-sub and stale-active discovery, org/user predicates and processing-vs-notification distinction. Stale-active must set forceDeltaResync and exercise actual bounded delta/importRecent, not just enqueue.
• Renewal discovery becomes a thin adapter to the primitive, not a competing PATCH/error writer. Preserve pending/error existing-active renewal-only admission: guarded PATCH, NEVER CREATE/CAS/catch-up. Retain the load-bearing renewal test's capability, replacing textual assertions with observable behavior. Keep revoked/archived exclusions and orphan/no-token expiration to prevent hot-loops.
• Keep ten-minute architecture, bounded batches/fair traversal and attempt versus processed watermarks; bounded zero-row cooldown without changing connector generation. Preserve per-connector isolation; enqueue/Redis failure cannot advance success watermarks. Fresh IDs survive retained completed/failed jobs; explicit recreate/catch-up payloads and three exponential attempts stay intact.

6. API demand-recovery adapter — ≤7 files / 600 lines

• apps/connectors-api/src/routes/connectors.ts; src/lib/{outlook-connector-guard.ts,outlook-connector-guard.test.ts}; existing test/{connectors-routes.test.ts,connectors-lib.test.ts,routes/outlook-access-token.test.ts,routes/outlook-user-tokens-delete.test.ts}, all under apps/connectors-api. Owner: API worker; guard regression budgets include disconnect/token routes, not only broad routing tests.
• Replace only access-token generic demand-heal with best-effort typed maintenance enqueue for requires_reauth. Preserve response, owner/org/token guards, 404/410 admission and 502 behavior; queue failure never discards a usable token. Keep explicit OAuth generic authorization behavior and narrow authorized-only revocation helper unchanged.
• Tests first in existing outlook-access-token.test.ts: cached return never authorizes/resets authorized_at; queued failure stays disconnected, package proof alone may heal. Preserve atomic disconnect/reconnect contracts and #400 history wrapper; auth replay/replacement regressions from slice 2 must still pass. Remove stale token-mint-heals comments, not other callers' semantics.

7. Truthful Outlook status on existing surfaces — ≤12 files / 950 lines

• apps/web/lib/connections/{outlook-oauth.ts,outlook-oauth.test.ts,outlook-gate-exemption.test.ts}; apps/web/app/connect-outlook/{page.tsx,page.test.tsx (new)}; apps/web/app/(settings)/connections/{page.tsx,page.test.ts,actions.test.ts}; apps/web/app/onboarding/apps/{page.tsx,page.test.tsx (new)}; apps/web/components/onboarding/outlook-onboarding-row.tsx; apps/web/lib/render-session-consumers.test.ts. Owner: web worker. Parents and existing rendering suites count in cap; no product actions.ts staging change.
• Replace duplicated OutlookConnectorRow interface with Pick<typeof connectors.$inferSelect, ...> including last_error; project and thread through actual gate, settings parent/OutlookRow and onboarding parent/row. Import the single browser-safe reason contract from slice 2. requires_reauth + null/unrecognized reason: “Checking your Outlook connection”, automatic retry and status reload only, NO OAuth form. Confirmed missing/dead/interaction marker: existing reconnect action.
• Keep gate CLOSED until authorized and let plain recheck/reload observe background recovery then redirect. The failed query parameter's “Try again” must not recreate an OAuth bypass. Preserve AuthScreen, Button, layout/tokens, sign-out, password/session/membership gates, explicit disconnect/archived/pending enrollment and unrelated error/other-provider semantics. No redesign, component hierarchy, new endpoint or schema.
• Render ACTUAL gate page, settings page/row and onboarding parent/row: guarded missing-grant detection → already-requires_reauth reason-only persistence → real reconnect CTA; transient → no new marker/no OAuth; unknown/null reason + failed query → no OAuth; provider-proven authorized → redirect on reload. Preserve reconnect pending/updated_at race regression in actions.test.ts; replace in-scope source-text gate assertions with observable access behavior, not re-pinned strings. Reuse existing renderToStaticMarkup pattern; new page suites only because these real surfaces have no existing direct rendering suites.

Regression-first acceptance matrix

Supplied preimplementation baseline: 43 tests (7 lifecycle, 19 processor, 17 renewal) passed old behavior, not recovery. Matrix below retains acceptance criteria; current execution evidence and its boundaries are recorded separately, not inferred from this baseline.

Explicit deletions and replacements for adversarial review

• Delete lifecycle challenge's direct requires_reauth patch and its tests expecting downgrade; replace with maintenance behavior. Keep webhook authenticity, null-owner guard, unknown events and batch isolation.
• Replace worker authorized-only guard with exact two-state eligibility, not no guard. Remove “any active row means both folders exist” branch/comments and tests asserting that shortcut; replace with coverage behavior.
• Replace create helper's unscoped active lookup and unconditional post-PATCH row update. Keep unique index, encryption, cleanup and all identity fences; do not delete rollback merely to simplify partial coverage.
• Remove competing direct renewal mutation/error branches only after shared primitive cutover. Preserve pending/error PATCH capability with observable tests, revoked/archived exclusions, orphan/no-token expiration, partial cleanup and per-connector isolation. Replace 404-only-expire with exact retirement then recovery-admitted CREATE; renewal-only still cannot CREATE.
• EXPAND, do not delete, scheduled authorized zero-sub/stale-active discovery to include requires_reauth/partial/expired coverage; replace permanent job IDs with fresh IDs. Preserve org/user predicates, attempt versus processed watermarks, fairness, enqueue-error behavior and actual stale-active delta. No coalescing machinery or lost late catch-up event.
• Delete API demand-heal generic patch and “mint proved credential alive” comments/assertions. Keep generic OAuth patch implementation, terminal downgrade guard in lib/connectors.ts, owner/token guard, and original authorized-only dead-grant revocation seam.
• Preserve revocation timestamps, never renewal proof. Replace unconditional OAuth display with reason-based UI; preserve closed access, enrollment/disconnect and unrelated semantics. Use schema-derived Pick. No helper/history duplication or alert-policy change.

Durable review dispositions

Accepted historical corrections (1–3, 5–11, 13): separate renewal-only admission; connector-before-two-subscription final locks; credential write/snapshot fencing; one in-memory rejected-bearer refresh across all cache paths; provider/token AbortSignal propagation and atomic Lua ownership operations (not atomic Redis/SQL); typed CREATE cause/rollback; renewable grant and source-typed absence; fresh queue IDs; exact well-known resource parsing; additive authorized discovery/404 retirement; truthful three-surface UI with guarded reason-only writer. These replace conflicting instructions above, not an additional parallel architecture. Historical Grok assertions are not final review authority.

Rejected/clarified: (4) ordinary requires_reauth reconnect already calls reopenConnectorIfTerminal before beginOutlookConnect in actions.ts:494–510, staging pending/updated_at; no redundant staging, keep its race regression. This does NOT fix refresh token-write races. (11) possible loss of authorized discovery is a clarification to preserve that capability, not an approved deletion. (12) OAuth lookback is 60 days, not the old instruction; worker fallback stays one day. Grok's 70-minute lifetime claim is wrong: default is 69.5 hours. Earlier “no external assets” claim was false: both docs contained Typekit and CDN tags; remove them without changing the global generator.

Accepted follow-up (Astra): negative credential fencing and known non-active reconciliation, including repeated-revoke fencing; regressions above. Existing architecture/caps unchanged.

Original implementation review dispositions (supplied): Astra's original eight findings fixed, including nine renewal-fixture failures. Accepted history-poisoning, negative-after-self-refresh, scheduling and dedup findings fixed. Unconfirmed OAuth CTA rejected because the user expressly forbids it. v1 renewal/lazy-backfill race rejected: crypto-v2 explicitly throws for v1; it is not a supported runtime. Malformed-ciphertext InfrastructureError is intentional, including cross-provider impact.

Final corrections verified clean: fresh Astra's three findings—acknowledged lease expiry, stale orphan retirement CAS and final-completion CREATE cleanup—fixed. Opus 5's long catch-up budget, missing-refresh distinction and stale comments fixed. Final Opus low found active confirmed-grant rows starving renewal; latest two-file correction uses exact canonical NULL-safe SQL before LIMIT, with actual-SQL SQLite batch-1 regression failing pre-fix and passing post-fix. Final Astra and Opus finders both CLEAN, no new actionable regression. Full reviews examined deletions and structural changes; last finder scoped only to the final predicate.

Accepted tradeoffs and deletions: fresh demand-token enqueue volume accepted; no coalescer losing late work. Obsolete patchConnectorStatus and source-pinned tests deleted; actual password/middleware guards unchanged. Recovery errorCodes intentionally map to other; no ledger/alert-policy change. Per-event DB fence cost accepted; per-page alternative rejected. These are not unresolved findings.

Implemented correction contracts and proof

Source-inspected: consumed-credential outcomes retain snapshot/context; renewal negatives use the shared writer; disconnect locks connector before token. Token persistence checks renewed lease/deadline around transaction lock waits/writes. packages/connector-outlook/src/subscriptions.ts:137–215,299–338 fences acknowledged lease expiry and keeps final CAS plus a separate five-minute post-authorization catch-up budget under the same renewed lease. Final-completion failures reach independent five-second own-CREATE rollback (:740–806). Orphan CAS proves continued absence and exact scanned fields, including NULL-safe last_renewed_at (packages/connector-auth/src/tokens/outlook-recovery.ts:180–253); missing refresh on a present row is not an orphan. Renewal SQL excludes exact confirmed reasons before LIMIT (apps/worker-mail/src/loops/outlook-subscription-renewal.ts:280–301). Pre-emit generation fences, bounded custody and stale-active dedup remain intact.

Supplied results: root lint passed, existing warnings/no errors; root typecheck 64/64 tasks plus scripts passed. Auth 122, connector-outlook 80, core 68, provider 108 + 1 skip, chassis 80 + 1 skip. Worker combined four-file run at 20:40 UTC: 4 files passed, 112 passed + 1 opt-in SQL skipped (113); replaces older 102 summary, not a whole-package total. Earlier focused renewal+processor 96 passed, separate scope. API 518 + 5 opt-in DB skips; web 223 earlier + 27 webhook-focused. Disposable SQL 18 passed/0 skipped after 113 migrations/provision; five lifecycle-flow scenarios with dependency doubles passed. Separate scheduler SQL 4 passed/0 skipped is not added to worker totals. Exact combined command is in the durable report; no rerun.

Separate scheduler SQL target: disposable 127.0.0.1:5440/app_outlook_recovery_verify_scheduler_20260909, created from template0, migrated and provisioned with login app. Actual queries passed the status × reason × coverage sweep. Exact-target validation with OUTLOOK_RECOVERY_SQL_TEST_URL absent: 3 passed, 1 skipped; eight invalid explicit URLs threw before connect. Cleanup removed the target database and two scripts totaling 108 lines; cluster roles, membership and global settings unchanged. Supplied evidence, not rerun by the docs worker.

Provenance: actual Opus 5 after Fable unavailable fallback (degraded:fable); Fable spend acknowledged, no amount or exact Opus cost available here. Astra substitution was user-directed after Grok was deemed untrusted. Main implementation high effort; small-fix writer medium and final finder low per workflow. Earlier token-read-comment/turbo mechanical changes are historical worker work; this final alignment edits only the three recovery docs and changelog, with no new behavior.

Limits: >1,000 history unresolved; running importRecent transport not cancelled but late emits fenced; Redis/SQL final-hook-to-COMMIT not atomic. No browser visual test: ss found no :5140 listener, no server started. No real Graph, session or live writes. Completed smoke cleanup removed the disposable database, two SQL scripts and two /tmp flow files. Exact boundaries and dispositions: durable report.

Verification and delivery gates

1. Run configured bun run test from web, worker-chassis, connector-outlook, connector-auth, provider, connectors-core, worker-mail and connectors-api directories. Focus via each package's Vitest script, never Bun's own runner on Vitest files. Include subscriptions.test.ts, all refresh-*.test.ts, outlook-access-token.test.ts, both scheduled-resync locations and the named UI rendering/OAuth/actions/gate suites; export/dep-isolation guards cover the new browser-safe subpath.
2. Exercise queue retention/late events and Graph/token interleavings with deterministic isolated fakes/Redis namespace. DB races require a disposable migrated/provisioned database under proper RLS roles per AGENTS.md, never shared dev/preprod, real sessions or endpoints. Render actual UI surfaces, then use ui-test on an authorized isolated local runtime; no dev-server launch or live mutations by a worker.
3. Keep regression tests only for plausible observable failures in the matrix. Shared-helper/API tests must assert outcomes, not field forwarding or source strings. A throwaway local smoke driver must exercise challenge→worker→provider test double→guarded status/delta end to end; no live Graph.
4. Root bun run lint and bun run typecheck, all new focused regressions and affected suites green before PR. Run cheap eval only if actual retrieval/routing/memory behavior changes; this plan does not change those contracts.
5. Two independent different-model adversarial reviews of implementation and deletions to convergence before PR. Worker reports commands/output and omissions; never commits. Orchestrator owns PR and CI-green gate. No merge/deploy without kwiss.

Open dependencies and stop conditions

#400 dependency is fulfilled on the user-reported merged base 7fa680c3. All seven recovery slices and subsequent corrections are implemented; final Astra and Opus finders CLEAN as scoped above. Metadata remains in-progress for lane acceptance pending; CI not yet available, pending, NOT MERGE-READY. Recovery is NOT merged/deployed; kwiss retains authority. Extra schema, durable lease/framework, alert-policy changes or over-cap work still require reviewed scope correction. Personal-owner paging remains prohibited. Residual limits are not claims of atomicity or mailbox delivery.

Generator caveat: future regeneration can reinsert Typekit/CDN assets; keep these documents self-contained.

Mailbox metadata and the completed operator watch remain in the design; neither proves current grant validity or mailbox delivery.