Continuous Clio source ingestion and explicit NetDocuments matter correspondence — design spec

Status: implemented on branch (PR authorized; not merged or deployed) · Author: cross-source-intelligence orchestrator (transcribed by a Fable seat) · Date: 2026-09-14 · Repo: north-os, branch feat/cross-source-intelligence · Related: NetDocuments MCP federation. No issue ID.

Intent

Agents must not rediscover implicit Clio-to-NetDocuments links each turn. North ingests the populated Clio sources continuously, records explicit NetDocuments correspondence as a bounded evidence cache with provenance, and binds it server-side in matter-scoped search. No graph, no ontology, no task storage while Tasks are empty.

Evidence

Read-only probes, 2026-09-13/14: 4,697 matters, 80,307 readable custom-field entries, zero redactions; 78,933 visible activities (77,598 time entries), 1,048 dated 2026-08-15 to 2026-09-14 inclusive; Tasks HTTP 200 with zero visible records; 81 distinct explicit workspace references, unverified as NetDocuments objects. Live NetDocuments samples: one document matched Matter = Clio source ID and Client = Clio client ID; a second matched Matter only. No global coverage claim.

A. Association storage: matter_netdocs_association

• A separate table, not app-owned keys in matter_metadata and not a privileged observation-write endpoint: a bounded evidence cache on an existing matter, not a graph. matter grants stay SELECT-only for app_role; no provider-namespace migration.
• One row per (organization, matter) with a composite org/matter FK, cascade on delete. Columns: up to three cabinet binding observations (cabinet id, discovered parent and child attribute names and slots, schema fingerprint), Clio matter and client source keys, the Clio connector id with its lossless authorized_at, the NetDocuments connection id, account and reach with its lossless authorized_at, checked-at, and historical evidence and coverage (state, matched keys, examined domains, partial/unknown bounds).
• Org RLS via pgPolicy + enableRLS; app_role gets narrow DML on this table only. No activity-table grants to app_role, scheduler_role or reporting_role.
• Reads derive availability from live connection posture, never from stored evidence, which grants nothing. Only organization-reach observations persist; personal observations stay request-local and never update, clear or invalidate firm rows.
• No serializable matter locks are promised (app_role cannot take them). Persistence uses conditional current-state checks on the row (both retained feeder identities and epochs, admitted posture, source keys, fingerprint). Cache reuse compares the retained identities and epochs against the current ones, not only pre/post checks inside one request; the observation is historical and any later source change makes it stale at read time.
• Current-lease and both-feeder epoch and posture checks run before and after remote work and gate rendering and persistence of returned results.

B. Clio provider metadata projection

• Keep the existing provider metadata key layout; add the authoritative Clio client source ID explicitly.
• One requested-field-set-aware writer serves poll, picker fetch, webhook, close/delete, conflict resolution and mail auto-track. A requested complete custom-field array replaces the whole array, cleared fields included; unrequested fields stay untouched; explicit null or redaction replaces prior readable content; unrelated app and fixture keys survive; omission is never written as null.
• Custom fields keep provider field IDs, types, picklist option IDs, soft_deleted and redaction markers. No firm field names, label dedup, or people/workspace identity merge.
• Historical enrichment is a separate metadata-only operation on existing eligible rows, not lifecycle catch-up: no creations, tracking or party changes, no watermark movement.

C. Activities: clio_activity

• New schema file packages/db/src/schema/clio_activity.ts. Rows unique on (organization, connector, source id); composite org-scoped connector FK; nullable composite matter FK; source type, date, note, raw quantity plus TimeEntry hours; provider actor and task refs as source IDs, not app employee identities; source updated_at and optional etag; local observation state. Field declarations follow the captured OpenAPI Activity_base (etag string, note string, quantity number). The non-time quantity unit is stored as provider-unspecified unless documented; seconds, hours or currency are never inferred. No prices, totals or rates.
• A visible projection requires an eligible matter. When a known activity's association becomes redacted or ineligible, the stored readable projection is invalidated and cleared, not skipped. Unknown ineligible payloads are never persisted.
• Eligibility: closed and tracking-archived matters stay eligible (existing behavior); disabled, untracked and provider-deleted are excluded; personal matter_hidden is not an ingestion exclusion.
• Absence from a completed full scan marks the row unseen with coverage unknown, never deleted, including on 404; there is no proven deletion authority.
• Reader: new packages/connector-clio/src/activities.ts; the capability on the connectors-core matter-sync seam returns pages, not row-only iterators. Extend ClioClient only for the needed GET activities forms, keeping origin, redirect, method, body and feature-switch guards. Cancellation threads through requests and retry waits; shutdown aborts and awaits active work. Activity rows are not exposed to agents in this slice.

D. Activity checkpoint: connectors.activity_sync_state

• One additive nullable JSONB column on connectors, not sync_progress and no checkpoint table. Typed contents: version; Clio authorization epoch; completed updated_since watermark; active scan (kind, lower bound, fixed start cutoff, scan id, exact provider continuation); last completed full scan; durable deduplicated per-matter backfill requests keyed by eligible persisted matter identity and source epoch, each with its own resumable cursor and range.
• Each page's upserts and checkpoint commit atomically; a failure advances neither, and committed pages resume. Only the terminal page advances the completed watermark, to the fixed scan-start cutoff. An invalid or expired cursor restarts the same fixed range idempotently; authorization errors do not restart forever.
• Each tick is bounded to five pages or 20 seconds, whichever first; the connector advisory lock is released between ticks; resumable batches never block webhooks.
• A newly tracked or re-enabled matter atomically enqueues a per-matter backfill request in the same column, served with the documented matter_id filter on GET /activities.json; no full rescan per track. Global and per-matter cursors resume independently; bounded batches interleave fairly so a global scan never starves newly tracked work; per-matter completion preserves the global completed incremental watermark. A request is removed only atomically with its terminal-page commit; a failure or re-track during an active pass leaves a later request pending. No queue framework, no checkpoint table; the periodic full scan serves only unseen/coverage reconciliation. Initial, incremental, per-matter and full coverage stay separate.
• Clio authorized_at becomes the epoch only after verifying every reauthorization writer bumps it. Each page transaction compares epoch and current source eligibility, locking source rows through sync_role; stale-epoch or cancelled pages never commit.

E. NetDocuments authorization epoch

• The connector updated_at generation is an operation nonce: reserveGeneration in apps/connectors-api/src/lib/netdocs-mcp.ts advances it on every lease issuance, so it would invalidate cached bindings on every search and reject concurrent readers. It stays unchanged for the credential service's CAS protocol and is never compared for observation validity. Neither the lease id nor the account/scope tuple is the epoch.
• The epoch is the existing authorized_at as lossless microsecond text, with connector, account and reach identity. The access lease exposes it from the same guarded connector state, read after any token rotation, as an additive one-release-compatible field; parsers and callers migrate. No separate unfenced app read, no generic generation column, no global auth framework.
• authorized_at must be monotonically fresh on every relevant path: the Clio general authorization writer in apps/connectors-api/src/lib/connectors.ts already sets it; the NetDocuments authorization and firm-transfer writers in netdocs-mcp.ts already bump it. Reach must not return to organization without a new epoch: the web owner-scope downgrade in netdocs-mcp-oauth.ts does not bump it by itself, so re-promotion must. The stageNetDocsConnector non-forcePending same-owner organization promotion in connectors.ts is named for monotonic epoch repair alongside the existing authorized writers; any other unguarded re-promotion found during implementation is fixed the same way.
• Matter mode refuses when the authoritative epoch is unavailable; generic behavior is unchanged. Cache and result rechecks compare the stable epoch plus current admitted posture, source keys and schema before results and persistence; a changed epoch requires rediscovery and never broadens access.

F. Deterministic bindings

• Discover complete accessible cabinet metadata with the raw parent-child lookup relationship, unique exact names and slots; enumerate candidate pairs without assuming Client/Matter labels.
• Probe the parent with the Clio client ID and the child with the Clio matter ID, with no user text or title filters. Compare raw attribute_values keyed by discovered name; slots only fingerprint.
• Several cabinets may legitimately match: record and search each uniquely corroborated cabinet, up to three. Competing pairs in one cabinet, incomplete discovery or exhausted bounds leave that cabinet unresolved; never choose the first.
• Bounds per invocation: six provider searches and three continuation pages, plus the existing deadline. Unexamined domains report as partial/unknown; an empty page with a next cursor claims no absence; a domain unresolved within bounds returns unresolved evidence, never a silent loop or restart.
• Revalidate the schema fingerprint before reusing a binding. UI evidence freshness is 24 hours with checked-at, with no claim of real-time remote schema detection. The workspace reference stays source-supplied and unverified.

G. Matter-scoped search

• Optional matter_id mode on the existing netdocs_search, on both strict surface schemas (packages/agent-runtime/src/tools/netdocs.ts and apps/mcp-server/src/tools/netdocs-search.ts); public args stay distinct from provider args. An explicit matter ID is required (no hidden MCP thread fallback); conflicting cabinet or attribute filters are rejected before any provider call.
• A dedicated association module loads the authoritative Clio connector identity and epoch (the posture enum alone is insufficient), resolves the eligible Clio matter and source keys, discovers or reuses current valid bindings, routes, and verifies. The connector parser is extended so exact dual-key comparison happens before lossy label/value truncation and the 20-attribute display cap. Identifiers are never trimmed or case-folded speculatively.
• Only exact verified matches render as matter-scoped hits; missing or mismatched hits count as unverified/partial, never as matter documents. Generic search stays unchanged.
• Continuations are bound to organization, actor/access identity, both feeder identities and epochs, matter and source keys, cabinet/schema binding, effective query and filters, and count; the exact provider continuation is carried unchanged. Cross-scope replay is rejected with bounded partial receipts.
• Both Clio and NetDocuments are guarded at lookup, result and persist. matter_id mode is advertised dynamically only when both feeders are admitted, because source-aware tool hiding is repo policy and the call-time guard does not replace it; stale or direct calls refuse. Generic netdocs_search remains available with NetDocuments alone.
• get_matter_context reports local status and routing only, never remote verification; its MCP context plumbing gains two-source posture checks.
• An explicit matter_id composes with research_handle instead of being refused: the scoped read reaches the typed matter-capable path, and the matter source survives canonical get and save replay and the existing escalation replay, which reuse the same read-input and provenance seam. Generic and provider argument schemas stay strict and unchanged.
• A scoped recorded read captures only the final actor-authorized hits that passed exact dual-key verification and were actually rendered, once at final settlement behind the closing feeder, matter, epoch and persistence fences, never during discovery or provider probes. Missing, mismatched and cap-truncated candidates contribute counts only.
• Recorded coverage stays per cabinet — returned, verified and rendered counts, missing and mismatched counts, reported totals and separate continuation tokens — rather than an invented exhaustive aggregate or one merged cursor. Unresolved, no-search and truncated results stay partial.
• Fresh replay of a recorded scoped read stays scoped: both feeder identities and epochs and the Clio source keys feed the scope digest, so a change on either side withholds the retained evidence even when the fresh hits are identical. Record expiry is unchanged.

H. Matter view

The existing MatterSummary, apps/web/lib/matters/{types,read}.ts, components/matters/matter-table.tsx and the embedded connection table use batched local firm-connection and source checks, never raw metadata, never a per-row remote fetch. A discreet status shows Profile keys matched, Reference only, Not checked or Unavailable or stale, with keyboard- and touch-accessible details: evidence scope, checked-at, explicit workspace-unverified distinction. No mapping editor or dashboard; DESIGN.md governs tokens and themes.

Verification

• Grants and isolation: actual scoped-role grants per table; cross-org FK and RLS rejection on both new tables.
• Metadata writer: removals, older replay, omitted versus null versus redacted, unrelated-key preservation, lifecycle invariance of historical enrichment.
• Activities: page replay, partial-page failure, an activity moved to a redacted matter, cancellation, reconnect ABA on the epoch, expense entries with provider-unspecified units and no price; a newly tracked matter with old activities uses only its scoped backfill without resetting the global watermark; per-matter and global retries and fairness.
• Epoch: two ordinary or concurrent lease mints leave an association usable; an actual revoke, reconnect or firm transfer invalidates it; a cached observation and a continuation both reject a Clio reconnect with unchanged source keys.
• Search: the both-feeder four-state matrix on both surfaces; no personal-to-firm persistence; raw attribute comparison beyond display limits; cursor cross-scope misuse; empty pages with continuation.
• UI: both surfaces, both themes, keyboard navigation, stale posture.
• Gates: isolated database migration and RLS smoke, the changed packages, root lint and typecheck, cheap eval gate because routing changed, synthetic-account ui-test on the authorized local runtime. No dev-server start, no live writes; source scripts and evidence stay outside the repo.

Routing and lane boundary

Config defaults: code Fable high, schema serialized on one Fable high worker. Reviews on the eventual diff: Astra high and fresh Fable high in Claude Code; RLS Sol max; deletions Grok xhigh. The study implementation is complete. PR creation is authorized; merging, deployment and worktree retirement still require explicit approval.

Implementation and verification (2026-09-14)

Implemented on feat/cross-source-intelligence with generated migration 0116_orange_spitfire. Independent reviews (Astra high, fresh Fable high, Grok deletions, Sol RLS) returned clean. The branch is prepared for pull request review; it is not merged or deployed.

• Implemented: matter_netdocs_association and clio_activity tables with org RLS and role grants; connectors.activity_sync_state; the requested-field-aware metadata writer with explicit clio_client_source_id and custom-field projection plus bounded historical enrichment; the paged activity reader and the worker activity loop (5 pages / 20 s ticks, atomic page + checkpoint commits, epoch fencing, per-matter backfills on track and re-track, periodic unseen reconciliation); the lease-exposed NetDocuments authorized_at epoch with the same-owner promotion repair; matter_id mode on netdocs_search on both surfaces with deterministic bindings, raw dual-key verification, scope-bound continuations and dynamic both-feeder advertising; the local correspondence block in get_matter_context; the matter-view status column with an accessible details popover.
• Gates on the final source: root lint and typecheck green (61 tasks); cheap eval gate — coverage 11/11, retrieval 27/28 (96.4% against the 90% threshold), router 4/4 gated with two non-gated boundary misses, memory recall 9/9 with a 48-memory environment warning. Package suites: agent-runtime 5,111 passed / 105 skipped, clio-sync 24 passed / 1 skipped, connector-clio 96, NetDocuments connector 204, worker-clio 92 (activity sync and backfill), MCP server 175, connectors-api 142 unit plus 6 Postgres, RLS 5 Postgres, persistence conflict 1 Postgres, web status 5.
• Database: migration generated from schema TS only, applied and provisioned on an isolated database; grants, RLS isolation, cross-organization FK refusal, the visible-requires-matter and organization-reach checks, app_role denial on matter UPDATE and on clio_activity, and cascade on matter delete were proved under the real roles, every write rolled back.
• UI: the status cell was mounted alone in a local headless browser with the compiled design-system stylesheet, both themes, keyboard (Tab, Enter, Space, Escape) and touch; no Next page render and no dev server.

Reconciliation with main (2026-09-15)

origin/main at 26e09860 (#423, direct-first research and resumable escalation) was merged locally into this branch, PR #431, and the scoped recorded-research fix was applied on top. The generated migration 0116_orange_spitfire is unchanged: no schema drift from the merge. Focused proof on 2026-09-15: 406 agent-runtime NetDocuments tests, 316 MCP-server tests and 269 NetDocuments connector tests passed, with both package typechecks green. Those counts are focused proof of the scoped recording path over typed fixtures; they are not reconciliation or CI evidence, and no full scoped model escalation was exercised. PR #431 carries the final reconciliation and CI evidence. The gate and suite counts recorded above are historical, measured on 2026-09-14 before this merge, and stand as written.

Operational limitations

• Clio-supplied NetDocuments workspace IDs remain unverified: the NetDocuments catalogue offers no workspace resolver, so the matter view names them as a source-supplied reference only.
• Firm-reach evidence persists in matter_netdocs_association; personal-connection observations stay request-local and never touch firm rows. Evidence never grants access — availability always follows the live connections, and a reconnect, revoke or transfer of either feeder makes retained evidence stale until the next matter-scoped search rediscovers bindings.
• No task corpus: Clio Tasks are empty and are not ingested. Activity rows are not exposed to agents in this slice.
• Activity ingestion and metadata enrichment are bounded and continuous (five pages or 20 seconds per tick, 25 enrichments per matter poll): recovery after an outage or reconnect is gradual, never a one-shot rescan, and a source absent from a full scan is marked unseen, never deleted.
• Not exercised: a live NetDocuments or Clio provider, live preprod writes, the real /matters and /connections/clio page renders.

Changelog notes (draft; the published changelog is updated only with a real commit)

• Matters: a discreet NetDocuments column shows whether a Clio matter's documents were matched by exact profile keys, with details on evidence scope and freshness.
• Assistant and Cowork: netdocs_search accepts a tracked matter and returns only documents whose NetDocuments profile matches the matter's Clio matter and client ids; the matter brief reports the local correspondence status.
• Clio: matter metadata now records the client id and custom fields continuously, and time and expense activity is ingested in the background with resumable checkpoints.
• Assistant and Cowork: a recorded research read can be scoped to a tracked matter; the record keeps only the profile-verified documents it actually returned, with per-cabinet coverage, and replays stay scoped to the same matter and connections.