## Summary
Adds nine deterministic QuickBooks financial marts to the existing mart_education
schema, published atomically by a single Redshift stored procedure that a dedicated
Lambda runner invokes. The runner is wired to the platform's existing
triggers.on_pipeline_success subscription to quickbooks-raw-sync, but success alone
is not treated as readiness — raw-sync intentionally reports successful checkpoint
outcomes. Before touching Redshift, the mart runner reads raw-sync's durable S3 state,
requires an idle active state with no pending sync or clean publication, fetches the
exact versioned completion manifest, and verifies its size and SHA-256. Only a complete
backfill or sync manifest is accepted; anything else returns a cheap skipped. The
accepted manifest's identity is pinned into mart lineage, and the procedure rejects any
required raw publication newer than the accepted completion. This is a build-and-
validate PR — legacy QuickBooks tables are untouched and no consumers (Aerie, Klair,
Core) are repointed; consumer cutover is explicitly out of scope.
The nine marts and their sources:
| Mart | Source |
|---|---|
| quickbooks_ap_transactions | raw_bill, raw_vendorcredit |
| quickbooks_bill_payments | raw_billpayment |
| quickbooks_bills | raw_bill |
| quickbooks_deposits | raw_deposit, raw_account |
| quickbooks_expense_transactions | raw_purchase |
| agg_quickbooks_financial_metrics | Same-refresh expense candidate |
| quickbooks_journal_entries | raw_journalentry |
| quickbooks_purchases | raw_purchase |
| quickbooks_vendor_credits | raw_vendorcredit |
Five non-deterministic legacy outputs (quickbooks_expense_reports,
quickbooks_pl_data, quickbooks_pl_monthly, qb_cost_opportunities,
qb_vendor_classifications) are deliberately excluded. All sources are assumed
populated by PR #759 under staging_education_quickbooks.
## Refresh architecture
quickbooks-raw-sync (Step Function SUCCEEDED, any outcome)→ EventBridge starts mart-education-quickbooks-refresh via on_pipeline_success
→ runner reads raw-sync durable S3 state
· requires idle active state, no pending sync / clean publication
· fetches exact versioned completion manifest, verifies size + SHA-256
· accepts only a complete backfill / sync manifest, else skipped
· skips a manifest already present in mart lineage (dedupe)
→ Lambda calls one Redshift procedure, pinning the accepted manifest identity
→ all nine marts publish atomically, or the whole refresh rolls back
- The refresh consumes the platform's standard success trigger — **no custom event bus,
no events:PutEvents, no bespoke on_data_ready trigger type.** Readiness is derived
and verified from durable state, not from event payload.
- backfill_progress, sync_progress, and skipped_overlap executions start an
inexpensive run that returns skipped; they cannot refresh marts from a mid-cycle
staging state.
- as_of_date defaults to the manifest's UTC completion date; an operator may override
it for a deliberate recovery run.
- At-least-once EventBridge delivery and manual recovery are safe: exact manifest
identity is recorded and deduplicated, so a manifest publishes at most once.
### Changes
Completion-manifest readiness gate *(new)*
- src/completion_state.py *(new)* — Reads raw-sync's durable S3 state, requires an
idle active state with no pending/checkpointed work, resolves the exact versioned
backfill_manifest / last_incremental_manifest, verifies size and SHA-256 against
the immutable reference, validates schema versions and extraction-id/key agreement,
and returns a verified CompletionManifest or an expected safe skip.
- src/handler.py — Loads the verified completion, short-circuits to skipped when
staging is not ready or the manifest was already consumed, defaults as_of_date to
the manifest completion date, passes the pinned manifest identity into the procedure,
and asserts the committed audit pins that exact manifest and timestamp.
- pipeline.json — Switches the trigger to on_pipeline_success, adds
UPSTREAM_STATE_S3_BUCKET / UPSTREAM_STATE_S3_KEY, and grants read-only
s3:GetObject/s3:GetObjectVersion on the exact state object and the production
quickbooks/manifests/* prefix only.
Stored procedure & lineage
- ddl/sp_refresh_quickbooks_financial_marts.sql — Adds seven completion-manifest
parameters, validates them, dedupes on manifest identity (concurrent duplicate call
returns without changing marts), rejects any required raw ledger publication newer
than the accepted completion timestamp, and writes the pinned manifest into every
audit row. Procedure signature widened accordingly.
- ddl/refresh_run_lineage.sql — Adds the seven NOT NULL completion-lineage
columns to the audit table.
- ddl/migrations/2026-07-21_completion_manifest_lineage.sql *(new)* — Adds the
seven completion-lineage columns (nullable, since historical rows predate pinning).
- ddl/migrations/2026-07-21_remove_unguarded_procedure.sql *(new)* — Drops the
obsolete two-argument procedure, applied only after the guarded signature exists so
there is no callable gap.
Marts, runner, reconciliation *(unchanged foundation of this PR)*
- Nine per-mart DDLs + agg_quickbooks_financial_metrics.sql, the warehouse-owned
naming migration, the Redshift Data API client (src/redshift_client.py) with
deadline-bound cancellation, the read-only legacy reconciliation tooling
(scripts/reconcile_legacy.py, RECONCILIATION.md), and owner registration.
Docs
- README.md — Documents the success-trigger readiness gate, the S3 state /
manifest verification, the skip semantics, and the ordered migration/deploy steps.
### What was removed vs. the initial approach
An earlier iteration of this branch published a bespoke Pipeline Data Ready EventBridge
event and added an on_data_ready CDK trigger type. That was replaced with the
durable-state readiness gate above. This PR therefore reverts all of it, and the net
diff against main contains no CDK changes and no quickbooks-raw-sync changes:
- Removed the on_data_ready trigger type, its EventBridge rules, schema, resource-name
helper, and CDK tests.
- Removed custom event publishing from quickbooks-raw-sync/src/main.py, its
events:PutEvents IAM grant, and its event tests.
### Design Decisions
- Success is not readiness. Because raw-sync legitimately succeeds on checkpoint
outcomes, the mart proves readiness from durable state and a checksum-verified
immutable manifest rather than trusting the trigger. Unready executions cost one cheap
skipped run.
- Manifest identity is the idempotency key. Both the handler and the procedure dedupe
on the exact (bucket, key, version_id, sha256), so at-least-once delivery and manual
replay publish each completion at most once.
- Fail closed on races. The procedure rejects any required raw publication newer than
the accepted completion timestamp, so a delayed trigger cannot publish against a later,
partially-checkpointed extraction.
- Atomic, all-or-nothing publication. All nine marts are replaced in one transaction,
each verified against its candidate before commit; any failure rolls the refresh back.
- Warehouse-owned naming. Columns follow warehouse conventions
(txn_date → transaction_date, doc_number → document_number,
credit_flag → is_credit, adjustment → is_adjustment, currency →
currency_code, qb_created_time → quickbooks_created_at, generic IDs →
bill_id/purchase_id/journal_entry_id/…); the aggregate is
agg_quickbooks_financial_metrics. The rename migration preserves existing rows.
### Known exceptions & limitations
- Expense account mappings 140/93 → 63210/63220 are hardcoded rather than governed
through a queryable mapping table.
- The procedure reads staging raw_* JSON because the clean projections do not
preserve every required field.
- Runtime uses the shared CQL_download_OM Redshift identity.
- Surtr deployment does not automatically apply warehouse DDL in a fresh environment;
migrations are applied manually in the documented order.
- Currency handling was intentionally not expanded.
- Consumer cutover is out of scope for this PR.
## Test Plan
- [x] Mart runner tests — 70 passed (readiness gate, manifest size/SHA-256 verification,
skip semantics, handler input/audit validation, Data API timeout cancellation,
reconciliation safety, read-only SQL enforcement, mart SQL contracts, atomic publication)
- [x] quickbooks-raw-sync tests — 83 passed
- [x] CDK tests — 641 passed / 17 suites (count dropped from 648 with on_data_ready
tests removed)
- [x] Ruff lint + format clean; CDK TypeScript build passes
- [x] Net diff against main contains no CDK or quickbooks-raw-sync changes; no
remaining on_data_ready references
- [x] Warehouse naming migration, DDL, guarded procedure, and one atomic refresh applied
to production Redshift; nine marts populated and reconciled with zero unexplained
differences (8 marts accepted expected_snapshot_drift;
quickbooks_expense_transactions accepted non_reconcilable_historical). No tables
dropped.
- [ ] Reviewer: confirm no legacy QuickBooks table or downstream consumer is repointed