Outlook automatic recovery

All seven slices implemented; final Astra and Opus finders CLEAN, no new actionable regression; NOT merged or deployed. Historical correction checkpoint 30dd5ad1, with subsequent fixes dispositioned in the report. Metadata stays in-progress for lane acceptance pending. CI is not yet available and remains pending; NOT MERGE-READY. Clean reviews confer no merge/deploy authority. Implementation plan · Durable verification and review dispositions. No authorization to change live state.

Latest product/evidence checkpoint: 68452506, reported pushed by the user; 30dd5ad1 is historical only. No deployment or CI result is inferred. Final cleanup also adds the existing-format changelog entry for this checkpoint; no product behavior changes.

Decision and acceptance

kwiss: “if you really are disconnected ... i don't want wrong case where we ask without reason”.

A Graph subscription challenge is maintenance, not proof that personal consent died. Automatically restore subscriptions with a renewable delegated grant, even when a cached access token is usable. Missing, revoked or genuinely interaction-required user grants still need action; no promise that every mailbox can recover without consent.

• Lifecycle maintenance, missing subscriptions, network errors, 429, 5xx and app credential/configuration failures never cause a new personal reconnect requirement.
• Only successful provider PATCH/CREATE establishing both required folders can heal an eligible requires_reauth connector. Cached token availability, timestamps and local active flags are not proof.
• Explicit disconnect/reconnect is fenced against stale work. Recovery admission is authorized or requires_reauth without an exact confirmed recovery reason; only recovery allows CREATE, authorization CAS and catch-up. Separate renewal-only admission preserves guarded PATCH of active subscriptions on pending/error and confirmed-reason requires_reauth; never CREATE, authorize or catch up there. Revoked/archived remain excluded. See residual concurrency limits below.
• No schema/migration, new queue, ledger, alert policy, UI redesign, unlimited mailbox history, live mutation or automatic rollout.

Baseline source mechanics — preimplementation

Historical repository-relative paths/line numbers below describe the inspected baseline, NOT current branch behavior. Verification was limited to supplied code seams, not new incident evidence.

• apps/web/lib/connections/outlook-lifecycle.ts:62–75: challenge directly patches requires_reauth; removal only revokes the subscription. Neither immediately renews or creates.
• packages/worker-chassis/src/mail-subscribe-queue.ts:52–119: existing identity tuple, forceDeltaResync, forceRecreateSubscriptions, first-connect lookback; three exponential attempts, completed jobs retained up to 24h.
• apps/worker-mail/src/processors/outlook-subscribe-processor.ts:109–146,197–284: authorized-only guard; any active row masks a missing second folder and bypasses delta unless explicitly flagged.
• apps/worker-mail/src/loops/outlook-scheduled-resync.ts:293–310,348–371,415–424: authorized-only zero-active/stale-active queries; scheduled payload omits forceDeltaResync and uses a permanently stable job ID.
• apps/worker-mail/src/loops/outlook-subscription-renewal.ts:308–322,376–408: active subscriptions with parent not archived/revoked (thus pending included); Graph 404 only expires the row.
• packages/connector-outlook/src/subscriptions.ts:124–165,392–432: per-folder lookup lacks connector_id and expiry; renewal obtains token, PATCHes Graph, then unconditionally updates local row by ID.
• packages/connector-outlook/src/subscriptions.ts:520–527 and apps/connectors-api/src/lib/subscriptions.ts:367–390 stamp last_renewed_at on revocation. That column cannot independently prove renewal.
• packages/connector-auth/src/tokens/refresh.ts:94–111,179–193,343–409: cached fast path; broad revoked exception class includes invalid_client, but narrow isDeadGrantError excludes app failures. Existing revoke anchor is authorized-only.
• apps/connectors-api/src/routes/connectors.ts:547–578: token-return path heals via generic patch after possibly cached token. src/lib/connectors.ts:732–742 resets authorized_at and errors: unsuitable as recovery writer.

One maintenance path

1. Web keeps verified webhook identity/client-state admission. Add a typed maintenance reason on the existing queue; challenge enqueues maintenance without connector downgrade. Removal first marks the matched local subscription revoked, then enqueues repair plus delta. Missed still requests delta. Enqueue failure remains visible to existing per-notification isolation; periodic discovery repairs lost enqueues.
2. Worker adapters re-resolve organization, connector ID, Outlook type, user scope and owner. Snapshot exact database generation before guarded credential reads. Jobs are hints, never authority; keep recovery and renewal-only admission distinct.
3. ONE maintenance primitive in connector-outlook owns ensure/create/renew serialization; ONE dedicated guarded recovery writer owns outcomes. Producers and workers are thin adapters, not second orchestrators. OAuth subscription creation and operator force-recreate use the same organization/connector lock and generation checks; pending enrollment is not recovery permission.
4. Require a renewable grant through loadTokens, which reads the user/provider token row including account_id inside the organization's RLS transaction: actual absent rows and absent refresh tokens yield typed missing-grant outcomes, not message matching. Distinguish malformed/decryption/config failures from missing consent. Ordinary token callers retain current cache behavior. Request https://graph.microsoft.com/Mail.ReadWrite; preserve shared union refresh scopes.
5. Parse exactly demonstrated me/mailFolders('Inbox'|'SentItems')/messages and users/<id>/mailFolders('Inbox'|'SentItems')/messages forms, anchored end to end with organization/connector/owner custody. Unknown resources, including opaque folder IDs not demonstrated by fixtures, are unresolved: neither healthy nor automatically deleted/recreated as a guessed folder. A duplicate-CREATE error is not proof without actual subscription reconciliation and encrypted client-state custody; no speculative resolver.
6. For each required folder, PATCH known provider subscriptions; CREATE missing coverage only under recovery admission. PATCH also reauthorizes; no POST /reauthorize within the prohibited ten-minute PATCH window. Expired/null-expiry coverage is not healthy. Graph 404 retires the exact old row under guard BEFORE folder CREATE to satisfy the unique index; renewal-only admission retires without CREATE. Provider default lifetime remains 69.5 hours.
7. For authorized/requires_reauth, before declaring a folder missing, validate exactly parsed/owned revoked/expired rows' retained provider ID and encrypted clientState custody; PATCH known ID under snapshot guards. 200 can establish real coverage; 404 retires exact row before CREATE. No repeated blind-CREATE or audit-row hard deletion. Capture/recheck last_renewed_at (null-safe exact DB string) alongside status/updated_at: cited revoke writers do not advance updated_at; repeated revoke during PATCH must defeat reactivation. Use existing metadata, no new writer/schema. Explicit disconnect and pending/error ACTIVE-only renewal remain unchanged.
8. Require real successful Graph PATCH/CREATE of BOTH Inbox and SentItems in this attempt before healing requires_reauth. Persist each result under row guards and capture its exact post-write generation. MailSubscription exposes only id/resource/expirationDateTime/changeType; proof requires valid future expiry, not local flags or last_renewed_at. Healthy authorized ordinary delta work need not gratuitously PATCH both folders; subscription proof is not delivery proof.
9. After network work, use the exact #400 withConnectorStatusTransition in one RLS transaction: connector lock first, then lock BOTH exact proven subscription rows in deterministic ID order. Verify organization/owner/connector/provider IDs, active status, unexpired coverage and captured post-write generations before connector identity/status/updated_at CAS. Subscription revocation need not advance connector generation, so connector CAS alone is insufficient. Zero matched rows is stale, never success; preserve authorized_at.
10. After ensured coverage and current recovery admission, run existing bounded delta/importRecent. Removal, 404/replacement, missing coverage and missed events request catch-up; stale-active scheduling sets forceDeltaResync. Preserve cursors, limits, deduplication and worker one-day fallback. OAuth alone supplies explicit 60-day firstConnectLookbackDays (outlook-oauth.ts:348–371); lifecycle/scheduled jobs never inherit it.

Concurrency and bounded retry

Implemented: bounded maintenance lease and deadline, owner-checked Lua extension/release, and real AbortSignal propagation through provider PATCH/CREATE/DELETE and token fetch. Token persistence checks the acknowledged renewed lease and deadline before the transaction, after lock waits and after writes. Redis ownership probes stay outside SQL transactions. SDK retry handling remains; no overlapping retry engine.

Every post-Graph subscription write locks/rechecks connector identity/generation first, then exact subscription status/provider ID/generation; recheck before subsequent Graph calls, final CAS and delta. Use exact Postgres timestamp strings, never Date round-trips; advance guarded generations with existing GREATEST(clock_timestamp(), updated_at + interval '1 microsecond') convention. Stale remote CREATE never activates a row: best-effort delete only IDs created by this attempt, preserving partial rollback and cleanup-failure warnings. Preserve original typed error/cause through OutlookConnectorError so bearer rejection, throttle and conflict remain classifiable.

Implemented credential fencing binds refreshed writes to the consumed account/timestamp snapshot: connector lock/generation first, credential lock/snapshot second. Disconnect uses the same connector-before-token order; deleted/replaced credentials reject replay, including OAuth replacement before a connector bump. Typed negative outcomes carry the consumed snapshot (or absence) and connector context; renewal failures use the shared negative writer. Ordinary reconnect retains pending/updated_at staging. No durable lease/schema or general auth framework.

ALL recovery negative writes (dead/interaction/missing, including reason-only) must atomically revalidate the exact consumed credential snapshot or continued absence, locking connector before tokens. Success-persistence fencing alone is insufficient. In typed recovery context, suppress/route refresh.ts's pre-persistTokens markOutlookConnectorRevoked side effect through this fence; never first invoke an unfenced legacy downgrade. Preserve ordinary callers' narrow helper/eligibility semantics and classifier.

Fresh BullMQ IDs preserve late demand/lifecycle work even with retained jobs; there is no outstanding-job coalescer. Scheduler queries retain authorized zero-sub/stale-active discovery and add unconfirmed requires_reauth/partial/expired coverage, excluding only exact confirmed recovery markers. Per-tick tuple deduplication prefers stale-active work so forceDeltaResync survives overlap. Attempt/processed watermarks remain separate; connector updated_at is not a scheduling watermark. Demand-token enqueue stays fresh per call, with accepted volume risk.

Final renewal discovery correction: apps/worker-mail/src/loops/outlook-subscription-renewal.ts:280–301 excludes requires_reauth with exact canonical confirmed reasons using NULL-safe SQL before ORDER BY/LIMIT. Active confirmed-grant rows cannot consume the earliest batch; null/unrecognized reasons remain eligible. This selection rule does not broaden the primitive's renewal-only capability.

Failure taxonomy and honest status

“Recovery pending” is not a database status. The browser-safe public @workspace/connectors-core/outlook-recovery export defines OUTLOOK_RECOVERY_REASON once: outlook_recovery_missing_grant, outlook_recovery_dead_grant, outlook_recovery_interaction_required. Transients do not overwrite confirmed reasons. Provider-proven success clears only these markers, preserving unrelated errors; reason-only writes share the identity/generation fence.

Minimal truthful UI, not redesign: replace duplicated OutlookConnectorRow with a schema-derived Pick including last_error, project it and thread through the gate, settings parent/Outlook row and onboarding parent/row. requires_reauth + null/unrecognized reason shows “Checking your Outlook connection” and automatic retry, NO OAuth form; a confirmed marker exposes the existing reconnect action. A failed query parameter's “Try again” must not bypass this distinction. Keep access CLOSED until authorized; a plain status recheck/reload, not OAuth, lets restored background state redirect/unlock. Preserve AuthScreen/Button/layout/tokens, explicit disconnect/archived/pending enrollment and other-provider/unrelated error-state semantics; no new component hierarchy, endpoint or schema.

API and health-lane alignment

Smallest safe demand-heal change: retain the existing access-token response and organization/owner/token admission guards, but replace the post-token generic authorization patch with best-effort typed maintenance enqueue when wasRequiresReauth. A cache hit is not proof; queue failure does not discard a usable token and the backstop remains responsible. Remove stale guard comments claiming token mint heals status; do not loosen pending/terminal admission.

Dependency provenance: kwiss reports PR #400 merged at 7fa680c3; this recovery branch is based on it. Earlier integration fast-forwarded ONLY fix/outlook-automatic-recovery from f30b280b to 457741e3d024a1bdcea7e519293c56beee6f11f1 (feat/connector-health, then 8 CI checks successful). That historical PR-open/merge-ready state is superseded, not current recovery CI evidence. Read-only source was /home/kwiss/workspace/north-os-connector-health. Recovery itself is NOT merged/deployed. Reuse the integrated helper; never copy it or reset authorized_at.

The integrated public withConnectorStatusTransition takes the transaction, organizationId/connectorId, optional errorCode and mutation callback. Call it inside the RLS transaction BEFORE status reads or FK writes: it locks the connector FOR UPDATE before invoking the callback. The callback applies identity/status/generation CAS and preserves authorized_at; omit errorCode on successful recovery. The wrapper returns the CAS result unchanged, emits zero events on no-op, records changed status in the same transaction, and rolls back on history failure. Never hold that transaction across Graph calls.

Use a dedicated recovery helper; preserve health changes to packages/connector-auth/src/tokens/revoke-connector.ts, packages/worker-chassis/src/fence/connector-lifecycle.ts and API generic patch. No duplicate ledger or generic patch recovery. Pages remain limited to access_scope=organization; personal Outlook access_scope=owner never pages. Existing app-failure signals remain operational, not personal consent requests.

Negative outcomes pass their reason as errorCode; the existing allowlist deliberately maps recovery reasons to other. Success omits errorCode. No ledger or alert-policy change; malformed ciphertext is intentionally an InfrastructureError, including its cross-provider impact. The per-event DB fence cost is accepted for safety; the per-page alternative was rejected.

Implemented boundaries and verification

The same renewed maintenance lease covers completion, final coverage CAS and catch-up; packages/connector-outlook/src/subscriptions.ts:137–215,299–338 fences acknowledged expiry and grants post-authorization catch-up its own five-minute budget. Worker outlook-subscribe-processor.ts:254–332 rechecks the authorized generation before each emit. Latest-known retired custody requires PATCH proof; active opaque resources and histories above 1,000 fail closed. No-active renewal-only returns before token access. Final-completion failure is inside own-CREATE rollback with an independent shared five-second budget (subscriptions.ts:740–806).

Final review disposition: CLEAN. Fresh Astra's three findings—acknowledged lease expiry, orphan retirement CAS and final-completion CREATE cleanup—are fixed and verified clean. Orphan CAS checks exact scanned generation/expiry and NULL-safe last_renewed_at under continued absence (packages/connector-auth/src/tokens/outlook-recovery.ts:180–253). Opus 5's long catch-up budget, missing-refresh distinction and stale comments are fixed; per-event cost and reason-to-other mapping are accepted. Final Opus low found renewal-loop starvation; the latest two-file predicate fix passed actual-SQL SQLite batch-1 regression after failing pre-fix. Final Astra and Opus finders both CLEAN, no new actionable regression. Full reviews covered deletions and structural changes; last finder scoped only to the final predicate.

Limits: histories above 1,000 remain unresolved; already-running importRecent transport is not cancelled, though late emits are fenced. Redis ownership and SQL's final-hook-to-COMMIT interval are not atomic. No browser visual check: supplied ss result found no listener on :5140; no dev server started.

Supplied proof: root lint passed with existing warnings/no errors; root typecheck 64/64 tasks plus scripts passed. Auth 122, connector-outlook 80, core 68, provider 108 + 1 skip, chassis 80 + 1 skip; worker combined four-file scope 112 passed + 1 opt-in SQL skip (113), four files passed at 20:40 UTC; earlier renewal+processor focused scope 96 passed, not additive. API 518 + 5 opt-in DB skips; web 223 earlier + 27 webhook-focused, separate scopes. Disposable SQL 18 passed/0 skipped; five lifecycle-flow scenarios with dependency doubles remain valid, not rerun. These are not CI approval or current mailbox delivery proof. Exact worker command and boundaries are in the durable report.

Separate scheduler SQL evidence supplied subsequently: 4 passed, 0 skipped on disposable 127.0.0.1:5440/app_outlook_recovery_verify_scheduler_20260909, created from template0, migrated and provisioned with login app. Actual queries passed the status × reason × coverage sweep. Exact-target validation with OUTLOOK_RECOVERY_SQL_TEST_URL absent: 3 passed, 1 skipped; eight invalid explicit URLs threw before connect. Target database and two scripts totaling 108 lines removed; cluster roles, membership and global settings unchanged. This separate result does not replace the earlier package total or close review/CI gates.

Review provenance: actual Opus 5 after Fable unavailable fallback, degraded:fable; Fable spend recorded without inventing an amount, no exact Opus cost available. Astra substitution was user-directed after Grok was deemed untrusted. Main implementation high effort; small-fix writer medium and final finder low per workflow. Historical token-comment/turbo mechanical work is separate from this final three-doc-plus-changelog alignment; no new behavior.

Case mapping: supplied metadata, not new evidence

Snapshot supplied as of 14:57:36; provenance: /tmp/outlook-pending-triage-facts/five-case-1788965855989.json and supplement-report.md; no independent current-state claim.

• lboone conn_BESPpEXv__uk5zg8: token updated 14:39, two active, last processed 14:47:48. mdollaway conn_1hQ6byh0niOx4VGb: token 14:39, two active, processed 14:39:12. iabehassera conn_x3tfUb45aDqaiTXO: token 14:10, two active, processed 14:39:12. All requires_reauth: attempt verified maintenance, not metadata authorization. Recorded processing does not prove the original Graph signal spurious.
• netdocstips conn_KEpQkOs84lSrd8D9: two active, token and processing September 7. Actual grant unknown; attributing this case to the stale scheduler is UNPROVEN.
• Fjalloh conn_zHnenlAUINPSbJpB: zero active/two revoked with future September 9 expiry, token and processing September 8. Exact revocation cause unknown; baseline renewal/creation eligibility deadlock verified before implementation, not a current-mailbox claim. Repair discovery includes eligible zero-row coverage.
• Operator's conn_mJ-MmJESPidG89R0: status-only pending→authorized approximately 14:35; authorized_at September 8 and updated_at 14:10 unchanged. Completed watch 14:43:49–14:53:49: exact consource.email.received count 0, final authorized, no recurrence observed. NOT delivery proof and not permission to recover other pending rows. Orchestrator read deploy-bot's final report; no further flips or monitoring.

Resolved review contracts and implementation stop conditions

1. Resource parsing is bounded to demonstrated well-known forms; unknown resource/conflict remains unresolved without provider reconciliation and client-state custody. No guessed-folder cleanup or opaque-ID subsystem.
2. Keep interaction-required/missing-grant outcomes typed and recovery-owned, with one in-memory rejected-bearer refresh retry and all-three-path scope checks; no shared revoked-classifier expansion.
3. Tests must establish exact connector/subscription/credential snapshot fencing, same-transaction final coverage locks, disconnect replay rejection and concurrent OAuth replacement protection. If the existing metadata cannot prove these invariants, stop for reviewed scope correction; never silently weaken safety or add a migration.

All seven slices implemented; final Astra and Opus finders CLEAN within the documented scopes. Lane acceptance and CI pending; CI not yet available, NOT MERGE-READY. Review dispositions and evidence: durable report. No recovery merge, deploy or current-delivery claim.