> [!IMPORTANT]
> Merge/deploy companion: [AI-Builder-Team/Surtr#2091](https://github.com/AI-Builder-Team/Surtr/pull/2091) (SURTR-1542). It applies the identical package latest-week tie-break (ORDER BY week_start DESC, id DESC) to Surtr's mart_education.sp_refresh_aerie_xo_contractor_package. The two must merge and deploy together; otherwise this repo's query and Surtr's mart pick differently on any future tie, and the shadow compare diverges. Deploy order for this PR: Convex first or together with the worker (see *Partial publishes* below).
> [!WARNING]
> Source freshness: the upstream ledger is about 3 weeks behind. As of 2026-09-29, SELECT MAX(week_start), COUNT(*) FROM staging_finance_xo.raw_contractor_invoices returns 2026-09-07 / 152,718 rows, so the latest invoiced week is 22 days old. Ingest is alive: on 2026-09-25 the max was 2026-08-31 with about 151k rows. Whichever read mode ships, contractor package rates and trailing-52-week totals will be only as current as this table. That also means neither side of the shadow compare can be fresher than 2026-09-07. It's worth confirming with the xo-contractor-invoices-refresh owner whether a ~3-week lag is normal XO invoicing lag or a stalled window.
## The live bug (fixed first, independent of the migration below)
sync/src/redshift/xo-contractor-identity.ts and xo-contractor-package.ts queried core_finance.xo_contractor_invoices_raw, a table that no longer exists in the warehouse. Surtr's seed-gateway-aerie.ts (dated 2026-08-31) flagged that Aerie's contractor sync should be repointed.
These two tasks are not behind LEGACY_EDUCATION_WAREHOUSE_READS_ENABLED (financial-worker's own comment: *"XO contractor tasks remain active"*). They run in production, and since the underlying table moved, every run has failed its query and published nothing. The failure wasn't silent: each tick returned status: "degraded" with error: query: …. The data just never refreshed.
The real, current table is staging_finance_xo.raw_contractor_invoices, and it has an exact column match for these queries. Both queries now point at it.
Verified against real production Redshift when this was first opened: the fixed queries return 3,059 identity and 3,026 package contractors. Surtr's mart_education.aerie_xo_contractor_identity / aerie_xo_contractor_package return 3,058 / 3,026. Surtr's procedures are a verbatim lift of these same queries, so near-equal counts are expected. The counts confirm the repoint reads the right ledger. They are not an independent cross-check.
pg output order changes in this PR. The identity query's alias LISTAGG now has a full sort key (see below), so the aliases array order differs from what the old query would have produced. No consumer has seen the old order recently: the pg path has published nothing since the table moved. The Convex alias index also re-sorts longest-first on read.
## The migration (F1/F2, matching A4's pattern)
Surtr exposes both marts as Gateway sources, aerie-xo-contractor-identity and aerie-xo-contractor-package, already granted to the live Aerie key.
- Gateway readers. redshift/xo-contractor-identity-gateway.ts / xo-contractor-package-gateway.ts are field-complete and Zod-validated. The identity reader drops rows whose aliases all trim away, exactly as the pg reader does.
- Payload shape. Every published row, from pg or the Gateway, is mapped field by field to a new shared wire contract, @bran/contracts/xo-contractor-sync. Package optionals are omitted when null, never sent as null. This matters because Gateway rows carry sourceRunId, and Convex's upsertXoContractor* validators reject undeclared fields. The first version of this PR would have failed every gateway-mode batch. The contract is locked from both sides:
- a chat test reads the registered mutations' own validators via exportArgs()
- sync tests check every record the worker sends
- a convex-test case proves a record carrying sourceRunId is rejected
- Lineage. sourceRunId is used only to validate and log, never published. gateway mode refuses a read whose rows span two mart publishes (mixed sourceRunId); that tick degrades and the next one retries. Otherwise it logs the one source run it published.
- Deterministic package pick (Keval's decision, 2026-09-29). When a contractor has several regular-payment rows in their latest week, the pick is now ROW_NUMBER() OVER (PARTITION BY contractor_id ORDER BY week_start DESC, id DESC): the highest invoice row id wins. week_start DESC alone tied, and Redshift breaks ROW_NUMBER ties nondeterministically, so the published rate could change run to run.
- A read-only check on 2026-09-29 found id unique and non-null: 152,718 rows, 152,718 non-null ids, 152,718 distinct ids.
- A before/after comparison changed no output: 3,046 rows under both orders, 0 rows in only one of them, and 0 contractors with a tied latest week.
- The Surtr side is the companion PR above.
- Partial publishes are reported, not hidden. Publication is not atomic: each 100-row batch is its own Convex mutation, and each mutation isolates every record's write. Both refreshes now publish through analytics/xo-contractor-publish.ts.
- records is the count Convex confirmed it wrote, never the attempted count.
- Any shortfall is an error naming it, for example PARTIAL publish: 200 of 3046 rows confirmed before this batch failed or Convex rejected 3 of 3046 rows. The tick degrades, and the next tick re-sends every row (upserts are idempotent per contractorId).
- To make per-record rejections visible, upsertXoContractorIdentity now returns { errors, total }, as upsertXoContractorPackage already did (XoContractorUpsertResult). This is a small Convex change that ships with this PR.
- Fail-closed: a batch counts as written only if its response acknowledges exactly that batch. A missing, malformed or wrong-total response counts as unacknowledged and is reported. Deploy Convex before, or together with, the worker; otherwise identity ticks report "unacknowledged" (degraded) until Convex catches up, although the rows are still sent.
- External error text (Gateway, Redshift, Convex responses) is flattened to one line and capped at 300 characters before it reaches a log line or tick summary (boundedErrMsg).
- Population floor. A read that "succeeds" with too few rows is refused before anything is published, in every mode (analytics/xo-contractor-population.ts). This covers the direct Redshift path, not just the Gateway: an empty read used to publish nothing and still report success with records: 0. The floor is 500 rows, the same as the Gateway readers' own guard, which stays in place; production has about 3,050 rows per entity. A read below it is an error, nothing is sent, and the tick degrades.
- Strict, all-or-nothing parsing, on both sides.
- z.coerce.number() turned null and "" into 0 and true into 1, and z.coerce.string() turned null into "null". An incomplete row could become a plausible contractor id, dollar amount or date.
- All XO readers (pg and Gateway, identity and package) now use the same parsers from redshift/xo-contractor-fields.ts:
- contractorIdNumber: a positive integer
- requiredFiniteNumber / nullableFiniteNumber: a finite number, or a string that is wholly a plain decimal (so 0x10 and 1e3 are rejected)
- isoDateText: the whole value must be an ISO date (optionally with a time part) naming a real calendar day, normalised to YYYY-MM-DD (so 2026-02-30 and 2026-01-01garbage are rejected)
- Optional team_name / company / currency use one nullableText parser, so "" is treated as absent on both sides; pg used to publish "" where the Gateway omitted it.
- source_run_id must be one printable, whitespace-free token (lineageRunId), because it goes into worker log lines.
- Monetary columns keep their sign. The ledger and the mart DDL put no sign constraint on them, and trailing_52w_paid_usd sums every invoice row.
- The pg readers now use parseRowsStrict instead of safeParseRows. One invalid row fails the whole read (the tick degrades and Convex keeps the last good snapshot) rather than silently dropping the row and publishing a partial snapshot as a success. The Gateway readers already worked this way.
- Shadow compare. analytics/xo-contractor-shadow-compare.ts compares field by field, keyed by contractorId. The pg side has no sourceRunId (it recomputes live on every call), so lineage is checked on the Gateway side only, as in A5.
- A failed or unclean shadow check is visible. In shadow mode, if the Gateway read or compare fails, or the check runs but comes back not clean, the rows are still published from Redshift. The refresh returns shadowIssue, which is PII-free:
- check failed: …, or
- not clean: N mismatched, N pg-only, N Gateway-only row(s), <lineage>
- incomplete: trailing52wPaidUsd not compared (pg day X vs mart publish day Y) when the run was clean but skipped the date-anchored field
The financial worker reports that tick as degraded, so a run without complete, clean parity evidence can't pass for a clean check. The compare result also counts duplicate contractor ids on each side. All shadow-side work sits inside one try, so nothing on the shadow side can block publishing from pg.
- XO_CONTRACTOR_READ typos are visible. An unrecognised value (e.g. gatewayy) still falls back to pg, so data keeps publishing, but the tick is degraded with a config: note.
- Deterministic aliases. The pg identity query now uses Surtr's mart ordering exactly: ORDER BY LENGTH(a.alias) DESC, a.alias ASC. LENGTH DESC alone left same-length aliases tied, and Redshift breaks LISTAGG ties nondeterministically. Aliases are compared order-sensitively, which is only safe because both sides now share that key.
- snapshotDate is not field-compared. It's a run-date stamp: pg gets the worker's CURRENT_DATE on every tick. The mart gets its own publish date, and Surtr's replay-idempotency guard deliberately leaves an already-published source run untouched, so that date can trail the worker's by a day or more. Comparing it would flag every row on any tick that lands on a different UTC day than the mart's last publish, which says nothing about data agreement. Surtr's own replay diff excludes snapshot_date for the same reason. Both dates are still reported as pgSnapshotDate / gatewaySnapshotDate.
- trailing52wPaidUsd is compared only when the days match. It sums a window anchored on that same CURRENT_DATE. When the two days differ it goes into skippedFields instead of producing a false mismatch.
- PII safety. Every compared field is redacted in a mismatch record. Canonical names, aliases and weekly pay are real compensation data tied to real people, so a mismatch shows which field and which id, and never a name or dollar amount in a CloudWatch log line. Dates are not redacted.
- Env gate. XO_CONTRACTOR_READ=pg|shadow|gateway (xo-contractor-read-mode.ts, documented in .env.example) defaults to pg. Both refresh functions share it, since they are the same migration object and always move together.
- Dry-run scripts.
- dry-run-xo-contractor-pg-fix proves the table fix against real Redshift. It exits 1 below the 500-row floor, so an empty read can't pass as confirmation.
- dry-run-xo-contractor-shadow runs a full shadow compare with PII-redacted output. It also prints both snapshot dates, any skipped fields and duplicate counts, and exits 0 only when the run is clean AND complete.
- Both scripts print only bounded error text. It uses await import() rather than a static import for credential-dependent modules, per the ESM/dotenv-ordering fix Mercy flagged on the A5 PR.
The existing regression-guard test that asserted the *old* table name has been fixed. It now asserts the new table and guards against regressing back to the dead one.
## Verified locally
- Real production Redshift: the table fix (counts above) and today's freshness query (warning at the top). Both were read-only.
- Live Gateway shadow dry-run, 2026-09-29. This ran dry-run-xo-contractor-shadow against real Redshift and the real Surtr Gateway. It was read-only: nothing was sent to Convex, and only counts and field names were printed.
- Identity: 3,078 / 3,078 matched, clean.
- Package: 3,046 / 3,046 matched, clean.
- Later runs used the strict, all-or-nothing, whole-value parsers, and no row was rejected on either side. They landed on a day when the pg date and the mart publish date were aligned (2026-09-29), so trailing52wPaidUsd was compared as well. The latest run reported "Clean and complete".
- Verbatim output is in the PR comments.
- Tests:
- sync: 90 files, 1,585 tests, all green
- @bran/contracts: 93 files, 1,259 tests, including the new contract test
- chat, touched files only: financialContractorPackage.test.ts (9 tests, including the validator-contract and upsert-result tests) and analyticsSyncSecurity.test.ts (18 tests)
- the full chat suite was not run
- Typecheck: clean across the workspace.
- Lint: clean. The only output is 2 warnings in chat/skill/forge-api/scripts/sindri.mjs, which this PR doesn't touch.
## Known limitations (not fixed here)
- Gateway mode still needs Redshift credentials. In financial-worker, both XO tasks are still gated on isRedshiftConfigured(), so gateway mode can't run without Redshift env vars. That's fine while pg stays the rollback; it needs revisiting before Redshift creds are torn down.
Linear: [AERIE-2489](https://linear.app/builder-team/issue/AERIE-2489/f1f2-fix-broken-xo-contractor-sync-gateway-parity-identity-package)
## Business Value
This fixes a production task that has failed on every run for weeks. Contractor identity and compensation data feeding Aerie's finance dashboards stopped refreshing when the source table moved; the worker reported degraded ticks, but nothing repointed it. That makes this a real correctness fix, not just migration progress.
It also lands the F1/F2 Gateway migration object, following the A4 pattern. Gateway mode can now actually publish, and the shadow compare is deterministic, so a clean shadow run means real parity rather than noise. Once shadow is confirmed clean, this sync can cut over to Surtr's Gateway the way School Directories already has, moving one more EC2 worker task closer to full teardown.
## Manual Effort Estimate
About 2 days by hand. Proposed; Keval to confirm or adjust.
- Diagnosing the break: trace which table Aerie queries, confirm it no longer exists, find the real table and its pipeline, and check column and shape compatibility.
- Fixing it: repoint both queries and validate against real Redshift.
- Migration half: Gateway readers, the PII-aware shadow compare, the env gate, two dry-run scripts and about 40 tests.
- Fix round:
- the shared Convex payload contract with validator-derived tests
- lineage validation
- alias tie-break alignment against the Surtr procedure
- date-aware compare semantics
- the population floor, strict numeric parsing and shadow-failure observability
🤖 Generated with [Claude Code](https://claude.com/claude-code)