The Trilogy Times

## Problem

The claude/openai/azure token-spend pipelines' pricing matcher (calculate_costs) silently set every cost column to $0 whenever a model had no row in core_finance.ai_spend_token_pricing. New models reached Redshift as $0-cost rows with no signal, corrupting the cost marts. Flagged by a pipeline observation on claude-token-spend-pipeline (claude-sonnet-5 / claude-fable-5).

Verified against prod (2026-07-07): claude-fable-5 = 538 rows / 57.7B tokens / $0 since 2026-06-09; claude-sonnet-5 = 121 rows / 13.1B tokens / $0 since 2026-06-30.

## Prevention (this PR)

calculate_costs now raises UnpricedModelError instead of substituting $0. Each handler completes as much as possible and reports the run PARTIAL (the existing run-status feature — amber, non-paging, resets the failure streak) so the gap surfaces without blocking priced data:

- claude — skip the unpriced row (no billed column); post-insert assertion flags any BU carrying tokens but $0 total (zero_cost_bus).

- azure — skip the unpriced token row (Azure's real spend lives in the separate cost_records table).

- openai — keep the row with $0 pricing-derived cost but preserve the real billed_cost_dollars from OpenAI's billing API (skipping would drop money data); report unpriced_models.

Tests: claude 104 / openai 101 / azure 70 passing; ruff clean.

## Data remediation (claude, operator-run)

features/surtr/ai-spend-pipeline/specs/06-claude-5-pricing-drift/ — authoritative rack rates + a two-window reprice runbook + verification SQL. The 3 pricing rows have been inserted in prod and the reprice kicked off alongside this change.

## Heads-up: openai has pre-existing pricing gaps

The scan surfaced 16 unpriced openai models (mostly historical; ~4 current, incl. gpt-image-2 at ~$4.3K real billed). Deploying the openai prevention makes its daily run report PARTIAL until those are priced — the intended honest signal, not a regression (billed cost is preserved). Pricing those is a follow-up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Problem

azure-ai-spend-pipeline has been recording partial_failure with 6–7 of 10 subscriptions dropped on most recent runs (07-03 → 07-06); their AI spend never lands in Redshift. The observer flagged these as per-entity-failure / retry-storm (HTTP 429).

## Root cause

Azure Cost Management throttles at the tenant scope, and the throttle window frequently outlasts a single subscription's per-request retry budget (~90s). The sweep processes subscriptions one at a time, each retrying independently. So once the tenant window trips mid-sweep, every *remaining* subscription 429s and is dropped in the same run — they fail as a batch, not individually. PR #254 (per-request Retry-After + a 5s inter-subscription pace) reduced the rate but can't outlast a multi-minute window, so the batch failure persists.

## Fix

Sweep all subscriptions, then run a focused second pass over only the throttle-failed subscriptions after a cool-down:

- Pass 1 — sweep every subscription (unchanged pacing).

- Pass 2 — retry only subscriptions whose failure looks transient (429 / rate-limit / retry-exhaustion) after AZURE_THROTTLE_RETRY_COOLDOWN_SECONDS (default 60s, env-tunable). By the time pass 1 finishes, the throttle window that tripped early on has usually reset, so the spaced retry recovers them.

- Hard failures (4xx / auth) are not retried — a retry won't fix them.

- Recovered subscriptions leave the failed set; only subscriptions failing both passes are recorded as failed / partial_failure.

The whole run stays well under the 900s Lambda timeout (a clean sweep is ~50s; the retry pass only runs when something was throttled).

## Testing

- 67 passed (Python 3.11, the Lambda runtime), ruff check + format clean.

- New test test_throttled_subscription_recovers_on_second_pass proves a subscription that 429s in pass 1 is recovered in pass 2 → clean success, no failed subscriptions, its spend inserted.

- Updated the two existing partial/all-fail tests to account for the retry pass.

## Residual limitation / follow-up

This recovers the common partial-throttle days. It cannot guarantee completion when the tenant is saturated for the entire run window (e.g. 07-05). The definitive fix is querying Cost Management once at management-group scope (1 call instead of 10) — deferred because it needs a Cost Management Reader grant at the MG scope (a permission change), tracked separately.

> Note: this PR is the Azure fix only. The observer severity change (missing-cost-data → CRITICAL) is intentionally kept in a separate change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## What & why

Every Google Chat alert the platform sends now also lands as a row in a tracker Google Sheet, so alerts can be triaged/owned/followed-up outside Chat. The two notifier agents create rows; the triage reconciler annotates the same row (it does not add a new one). A few manual columns to the right are never touched by any automation.

Invariant: one SNS message == one chat card == one appended row. Because the notifiers only publish when a card is actually sent (failures always; partials/observer after their dedupe/throttle gates), a throttled or skipped alert writes no row.

## Row lifecycle

FAILED / PARTIAL / CRITICAL / WARN  ─▶  gsheet-tracker Lambda appends a row (cols A–K)
(SNS notification topic)                       key = run_id (col F)
│
triage reconciler (every 15 min)    ─▶  fills Triage Issue (L) + Fix PR (M) on the SAME row
learns Issue/PR from GitHub                            │
humans fill manual columns (N+) ── never touched by automation

## Column layout (unified tab)

| Col | Field | Written by |

|-----|-------|-----------|

| A | Logged At (UTC) | tracker Lambda |

| B | Type (FAILED/PARTIAL/CRITICAL/WARN) | tracker Lambda |

| C | Environment | tracker Lambda |

| D | Pipeline ID | tracker Lambda |

| E | Pipeline Name | tracker Lambda |

| F | Run ID (correlation key) | tracker Lambda |

| G | Detail (error / summary, truncated) | tracker Lambda |

| H | Link | tracker Lambda |

| I | Score (observer) | tracker Lambda |

| J | Consecutive Failures | tracker Lambda |

| K | Partials Since Last | tracker Lambda |

| L | Triage Issue | triage reconciler |

| M | Triage Fix PR | triage reconciler |

| N+ | your manual columns | humans only |

The Lambda appends only A–K; the reconciler updates only L{row}:M{row} on the matched row → N onward is guaranteed untouched.

## What changed

1. New gsheet-tracker Lambda (pipelines/cdk/lambdas/gsheet-tracker/)

- Subscribes to the existing SNS notification topic with a filter policy for pipeline_failure / pipeline_partial / observer_alert; appends A–K.

- Reuses the google/service_account/gdrive Secrets Manager credential (adds the spreadsheets write scope). Lazy gspread import; src/requirements.txt per the repo's PythonFunction bundling rule.

2. Observer → notification topic (Surtr/src/lib/tracker/publish.ts, wired in observer/index.ts)

- Publishes an observer_alert at the same gate as the observer's GChat card. gchat-notifier already ignores observer_alert, so no double-post.

- Injected unconditionally (SURTR_NOTIFICATION_TOPIC_ARN) so alert-tracking is independent of the triage feature flag.

3. Triage L/M writeback in the reconciler (pipelines/cdk/lambdas/triage-reconciler/)

- The triage GitHub Actions workflow has no AWS creds, so the sheet update runs server-side in the reconciler Lambda instead — the component that already polls GitHub for the Issue/PR state and has creds. It fills L/M on the row matched by run_id (from recent_run_ids), best-effort. No GitHub Actions secret, no repo variable — reuses the AWS google/service_account/gdrive secret.

## Design notes

- Centralized, server-side writes: both sheet writers are AWS-cred'd Lambdas; the lean stdlib gchat-notifier and the credential-less triage runner are untouched. No service-account JSON is copied into GitHub.

- The sheet id is a non-secret default in the stack (access is controlled by sharing the sheet with the SA), overridable via context/env — so it deploys live with no manual step.

- Every write path mirrors postObserverAlert: never throws, never blocks the notifier / observer / reconciler.

## Testing

- Lambda: 251 pytest green (new: row-mapping + gating for the tracker; _from_item list decode, _sync_tracker, and tracker_sheet.update_row for the reconciler; all never-raise).

- Surtr: tsc clean, biome clean, 14 vitest green (new tracker publisher + observer trigger).

- CDK: tsc clean both apps; pipeline-shared-stack synth test green (Lambda count 8 → 9).

- ruff check + ruff format --check clean.

## Go-live

No manual config. On merge, the normal CD deploys both stacks:

- pipelines/cdk → tracker Lambda (append A–K) + reconciler (fill L–M) go live (sheet id baked; secret grant in place).

- infra (surtr-app) → observer starts publishing observer_alert.

Already done: the sheet is shared as Editor with netsuite-wrapper@netsuite-wrapper.iam.gserviceaccount.com, and the header row A–M exists. (Note: the reconciler fills L/M on its 15-min cycle, so those cells populate within ~15 min of the Issue/PR opening.)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Problem

The pipeline observer computes verdict deterministically from finding severities: score = 100 − (C25 / H10 / M4 / L1), then ≥90 OK / 60–89 WARN / <60 CRITICAL. A single Critical-severity finding nets score 75 → WARN. So when the observer correctly flagged "zero billing records fetched and inserted" on gcp-billing-pipeline as a C finding, the run only surfaced as a WARN — and it sat mis-triaged for 16 days while GCP spend was missing from the dashboard. The Claude-token-spend gap was similar: new models absent from the pricing config (costs stored as $0) were classified H (schema-drift) → score 86 → WARN.

## Change

Make missing cost data surface as CRITICAL:

1. evaluate.ts::computeScoreAndVerdict — any C-severity finding forces verdict = CRITICAL, independent of the numeric score. A "Critical"-severity finding netting only a "WARN" run verdict was incoherent; this aligns the verdict with the severity. Score is still computed and shown.

2. prompt.ts rubric — missing / zero / stale / undercounted cost, spend, billing, usage, or pricing data is a C-tier finding, even when the mechanism would otherwise be High (a $0 price from schema-drift; a dropped account/subscription from per-entity-failure; zero billing rows from empty-but-success). Updated the calibration block and worked Example 6 to match.

3. OBSERVER_VERSION 1 → 2 so cached observations re-score under the new rules and the dashboard reflects the escalated verdicts. The existing hadPriorObservation gate ensures the backfill re-score does not re-fire alerts — only net-new runs alert (now as CRITICAL).

Alerts (GChat + SNS triage) already fire on CRITICAL, so no alerting change is needed — the escalation flows through automatically.

## Testing

- 230 derive + lib tests pass, tsc --noEmit clean, biome clean.

- Flipped the test that pinned the old contract (a single C-tier finding lands in WARN) to assert CRITICAL, and added one asserting any C dominates the verdict regardless of score band. Test fixtures' observerVersion pinned to the OBSERVER_VERSION constant so they don't rot on future bumps.

## Scope note

Deterministic C → CRITICAL applies to every genuine data-loss finding (empty-but-success, truncate-without-load, outcome-integrity, dry-run-leakage), not only cost pipelines — a Critical finding should read as a Critical run. The cost-data rule additionally ensures spend gaps get tagged C in the first place.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## What

A key-gated REST layer over the finance_dw Redshift warehouse at /api/v1 on the klair-mcp service, plus a runtime-neutral SKILL.md that agents install once and never need updated when the API changes. Replaces the blocked direct-psql-credentials path for agent access to raw warehouse data.

## How it works

Access & auth

- API keys in DATA_API_KEYS env (name:key,..., same shape as SERVICE_TOKENS), timing-safe comparison, accepted as Authorization: Bearer or X-Api-Key

- Every request logged with the key holder's name (audit trail); 50 req/min sliding-window limit per key

- Mounted before the Clerk/MCP auth stack — API keys never touch the OAuth flow; existing MCP behavior untouched

Endpoints

- GET /api/v1/schemas, .../{schema}/tables, .../{schema}/tables/{table} — discovery

- POST /api/v1/query — SELECT/CTE only (existing hardened validator); SQL runs as written, no imposed row limit, optional opt-in limit; 5-min statement timeout (DATA_API_QUERY_TIMEOUT_MS); real Redshift errors surfaced so agents self-correct

- GET /api/v1/meta (key-gated) + GET /api/v1/openapi.json (public) — the self-describing surface, both generated from a single ontology module (data-api-ontology.ts) with per-endpoint statuses, auth/rate-limit/error conventions

- GET /skill (public) — zips the live skill directory for install

Skill version signal

- Content hash of the skill pack stamped into the SKILL.md frontmatter + a generated server module (npm run gen:skill-version)

- X-Skill-Version header on every Data API response; /meta reports skill: {version, install}

- Installed skills self-report stale (note-once rule, never refuse to work)

Guard tests (the piece that keeps the self-describing surface honest)

- Contract test: router routes ≡ ontology endpoints, statuses complete, public/gated security correct

- Sync test: committed version stamps ≡ fresh content-hash recompute

## Testing

- 1066 tests pass (31 new: auth middleware, ontology contract, version sync)

- Live-verified against the compiled production build (node dist/server.js): auth accept/reject, SELECT-only enforcement, no-limit passthrough, /meta, /openapi.json, /skill zip, X-Skill-Version on all responses incl. 401s

- Negative-tested the sync guard: skill edit without regeneration fails the test

- Not verified: live Redshift queries (no warehouse creds locally) — needs a deployed smoke test with a real key

## Deploy notes

1. Mint keys: echo "rsk_$(openssl rand -hex 32)"; add DATA_API_KEYS=name:key,... to the ECS task definition

2. ./deploy.sh

3. Smoke-test the four data endpoints against https://mcp.klair.ai with a real key

4. Point users at GET /skill + hand them a key

Note: this is deliberately raw access under the single privileged MCP_user — every key sees everything that DB user can read; attribution is via logs, not grants.

🐦‍⬛ Generated by a very good bot

<!-- CURSOR_AGENT_PR_BODY_BEGIN -->

## Summary

Adds a default-on Mercy watcher to drones run so the harness auto-addresses Mercy's post-addresser findings without a human standing by. New src/mercy-watcher.ts module contains the pure parseMercyVerdict (unit-tested at every band) and the bounded watchMercyOnPr loop (round cap = 2, two-strikes anti-thrash, CI + Mercy-check settle between rounds, never-throws degrade to structured absent/parked/blocked outcomes). CLI kill switch: --no-mercy-watch. When the watch is off OR Mercy is absent, the existing auto-fire path is byte-identical to the pre-AI-129 shape.

## Why it's needed

The 2026-07-06 dogfood needed 5 manual Mercy-address fires across #61/#63/#64/#65 — an operator noticing Mercy findings, hand-firing drones address --review-id, waiting for the re-review, and repeating. Automating that loop closes the last "human notices, human re-fires" gap on the auto-fire pipeline and mirrors the CI-watcher / --max-review-loops pattern already in place for the drone-reviewer's own verdict.

## Changes

- src/mercy-watcher.ts (new) — parseMercyVerdict(reviewBody, state) returning {critical, warning, info, total, blocking, state, explicitZero, parsed}; watchMercyOnPr(...) bounded loop; renderMercyWatcherOutcome + renderParkCommentBody for operator surfacing; injectable fetchers/runners for unit-testability. Never throws — every error path routes through a structured MercyWatcherResult.

- src/mercy-watcher.test.ts (new) — 29 tests covering the parser at every band (0-findings, N-findings, withheld-COMMENTED-with-0-findings, CHANGES_REQUESTED, unparsed fallbacks, whitespace/case drift), the bounded loop (first-read clean, round-1 clean, exactly-at-cap park, two-strikes park, Mercy-absent, initial-fetch-throw blocked, re-fetch-fail park, owner/repo-unparseable blocked), the two-strikes fingerprint gate, and the AI-129 header-separator normalizer.

- src/addresser.ts — widened extractExpectedFindings header regex to accept ·/-/—/– as sev/dim separators (whitespace-anchored, so commas inside multi-token severities like High, Medium don't get mis-split). Exported for testability. Mercy's bold-head shape now flows through the same AI-127 severity/dimension cross-product cross-check as the drone reviewer's · shape.

- src/events.ts — new event types mercy_watcher_started, mercy_address_attempt, mercy_watcher_completed (parallel to ci_watcher_* so dashboards can attribute Mercy-address rounds distinct from the drone-reviewer-address loop).

- src/runner.ts — invoke watchMercyOnPr after the auto-fire loop + CI watcher. Exported isMercyWatchEnabled(mercyWatch) helper as the pinned kill-switch gate (byte-identical to mercyWatch !== false but pinned by a table-test).

- src/runner.test.ts — 3-case truth-table pin on isMercyWatchEnabled (undefined/true/false).

- src/cli.ts — --no-mercy-watch kill switch, --mercy-max-rounds <n> (default 2, max 5), --mercy-absent-timeout-sec <n> (default 300s).

- BACKLOG.md / ROADMAP.md — AI-129 flipped to IN-FLIGHT with the design decisions logged (default-on, round cap 2, exit on finding count not APPROVED, two-strikes, never-throws).

### Contract surface affected

- RunDroneInput in src/runner.ts: added optional fields mercyWatch?: boolean (default true), mercyMaxRounds?: number (default 2), mercyAbsentTimeoutSec?: number (default 300). All three are optional — existing callers see no shape change. The only behavior change on a call that omits these fields is that the Mercy watcher runs by default; suppressing with mercyWatch: false restores the byte-identical pre-AI-129 shape.

- Consumers: src/cli.ts (auto-populates from --mercy-watch/--mercy-max-rounds/--mercy-absent-timeout-sec flags), src/runner.test.ts (new gate test).

- DroneEvent union in src/events.ts: added MercyWatcherStartedEvent | MercyAddressAttemptEvent | MercyWatcherCompletedEvent.

- Consumers: current downstream readers ignore unknown event types (schemaVersion 1); no code updates required outside the runner + watcher that emit them.

- extractExpectedFindings in src/addresser.ts: newly exported. Regex widened to accept ·/-/—/– as sev/dim separators. The canonical drone-reviewer Critical · security-review shape parses byte-identically to before (pinned by the "no-regression" test in mercy-watcher.test.ts).

## Breaking changes

None. The Mercy watcher is a clean add-on: default-on but always killable (--no-mercy-watch), always bounded, and never throws (any error degrades to a structured absent/parked/blocked outcome). When the watch is disabled OR Mercy is absent within the bounded wait, drones run behavior is byte-identical to the pre-AI-129 shape — pinned by the isMercyWatchEnabled truth-table regression test in src/runner.test.ts. The addresser header-separator widening is a strict superset of the prior regex (canonical · shape unchanged).

## Test plan

- [x] pnpm typecheck → clean (no errors).

- [x] pnpm exec vitest run → 38 test files, 724 tests passed (up from 695 pre-AI-129).

- [x] pnpm exec vitest run src/mercy-watcher.test.ts → 29 tests passed (parser bands, bounded loop, park, absent, blocked, two-strikes, kill-switch regression, renderers).

- [x] pnpm exec vitest run src/addresser.test.ts → all 60 existing addresser tests still green after the header-separator widening (no regression on the canonical · shape).

- [x] pnpm exec vitest run src/runner.test.ts → all runner tests green including the new 3-case isMercyWatchEnabled truth-table pin.

- [x] pnpm test → full suite green (vitest + scripts/run-python-tests.mjs).

- [ ] Reviewer-side (live): fire drones run --task tasks/.../<spec>.md against a small change; verify (a) after the drone opens the PR + reviewer/addresser converge, the Mercy watcher stderr line appears ([drone:mercy] Starting Mercy watcher on PR #N…); (b) with Mercy present + 0 findings, exits clean and no addresser rounds fire; (c) with Mercy present + N>0 findings, fires ≤2 addresser rounds and either converges to clean or posts a park comment; (d) with --no-mercy-watch, no [drone:mercy] lines appear at all.

## Verification artifact

Local run of the new watcher tests:

✓ src/mercy-watcher.test.ts (29 tests) 1291ms
✓ parseMercyVerdict — parses canonical N-finding line
✓ withheld COMMENTED with 0 findings is CLEAN — the load-bearing boundary case
✓ CHANGES_REQUESTED with critical findings is blocking and not-clean
✓ APPROVED with 0 findings is CLEAN (never gate on approval)
✓ falls back to state when body carries no finding-count line
✓ extractExpectedFindings — still parses canonical drone-reviewer '·' separator (no regression)
✓ extractExpectedFindings — parses Mercy's hyphen-separated bold head
✓ extractExpectedFindings — parses em-dash and en-dash separators
✓ extractExpectedFindings — falls through to severity='unknown' when no bold head is present
✓ findingsAreSubset — two-strikes fingerprint gate table-test
✓ watchMercyOnPr — exits CLEAN on first read when Mercy verdict has 0 findings — no addresser fires
✓ watchMercyOnPr — exits CLEAN after round 1 when Mercy re-read shows 0 findings
✓ watchMercyOnPr — PARKS at the round cap and posts a park comment (boundary: exactly-at-cap)
✓ watchMercyOnPr — PARKS on two-strikes when the round-2 findings are a subset of round-1's (no re-fire)
✓ watchMercyOnPr — returns ABSENT when Mercy never posts within the bounded wait
✓ watchMercyOnPr — degrades to BLOCKED when the initial mercy-fetch throws — never infinite-loops, never aborts
✓ watchMercyOnPr — PARKS when Mercy re-fetch fails after round 1 (never throws / never infinite-loops)
✓ watchMercyOnPr — degrades to BLOCKED when repoUrl doesn't parse as owner/repo
✓ renderParkCommentBody surfaces residual verdict + rounds
✓ renderMercyWatcherOutcome tags severity per outcome kind
… 9 more
Test Files  1 passed (1)
Tests  29 passed (29)

Full suite: pnpm exec vitest run → 38 test files, 724 tests passed in 3.91s.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a href="https://cursor.com/agents/bc-7e22cc01-95da-4eb6-82bd-092a96f223af"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-7e22cc01-95da-4eb6-82bd-092a96f223af"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

## Summary

- Add Admin → Monitoring & Alerts as a central place for admins to review system job health and configure scheduled reports/reminders.

- Add a definition-centric Monitoring model:

- configurable product definitions for scheduled reports/reminders

- read-only system jobs for code-owned platform crons

- one consolidated monitoringRuns ledger for all execution history

- Add configurable definitions for:

- Buildout Freshness reminder

- REBL3 DD reconciliation report

- Add detail routes at /admin/monitoring-alerts/[definitionKey] for configuration, run history, errors, trace IDs, and manual runs.

- Add a generic Monitoring scheduler that runs due configurable definitions on 15-minute schedule slots.

- Keep code-owned system jobs visible with health, schedule, recent runs, errors, trace IDs, and no config controls.

## Why

Aerie has scheduled reports, reminders, syncs, maintenance jobs, queue processors, and data-changing automations, but admins did not have a single place to see whether those jobs are healthy or to adjust the report/reminder settings that were previously hardcoded or env-backed.

This PR creates the Monitoring & Alerts foundation around user-facing definitions rather than raw cron rows. That lets admins reason about “the REBL3 DD reconciliation report” or “the Buildout Freshness reminder” even when the implementation uses scheduler ticks, dynamic recipients, or multiple send times.

It is also the first building block toward fully customizable, ad-hoc alerts and monitors: the definition, recipient, schedule, run-history, and scheduler foundations introduced here are intentionally shaped so future platform-defined or user-created alert definitions can plug into the same model without bespoke UI each time.

## Key product decisions

- Monitoring home is split into:

- Configurable reports & reminders

- Read-only system jobs

- Configurable reports/reminders are DB-backed definitions.

- Read-only system jobs remain code-owned in the registry to avoid DB/code drift.

- REBL3 DD reconciliation is modeled as one report with multiple send times, not separate morning/evening products.

- FO Buildout Deferral remains configured only in Admin → Automations; Monitoring does not duplicate that configuration.

- The old Buildout test-recipient override is superseded by editable recipients plus Run now.

- Missed schedule slots are skipped; the scheduler does not catch up old sends.

## Configurable reports/reminders

### Buildout Freshness reminder

- Defaults from current env-backed behavior:

- enabled from BUILDOUT_ACCOUNTABILITY_EMAILS_ENABLED

- To: Site P1 DRI

- CC: BUILDOUT_ACCOUNTABILITY_MANAGER_CC_EMAIL

- Supports dynamic recipients where relevant:

- Site P1 DRI

- Site P2 DRI

- Sends one email per Buildout attention group using the configured recipient rules.

### REBL3 DD reconciliation report

- Defaults from current env-backed behavior:

- enabled from REBL3_DD_MONITOR_ENABLED

- To recipients from REBL3_DD_MONITOR_RECIPIENTS

- send times 08:00 and 17:00 America/New_York

- Starts fixed-email-only for recipients.

- Keeps its existing reconciliation receipt/run evidence and email behavior, including visible success/no-email outcomes when there is nothing to send.

## Schedule and recipient editing

Configurable definitions support:

- enabled/disabled state

- fixed timezone: America/New_York

- daily, weekly, or monthly cadence

- multiple weekdays for weekly cadence

- one monthly day with last-day fallback

- multiple send times

- 15-minute send-time precision

- separate To and CC recipient lists

- raw fixed email recipients

- contextual dynamic recipients where the definition supports them

Initial settings are seeded from existing env values. After the DB row exists, DB config is the source of truth.

## Runs, health, and diagnostics

- monitoringRuns records all execution history:

- system job executions

- Monitoring scheduler ticks

- scheduled report/reminder sends

- manual report/reminder sends

- future alert evaluations/runs

- Each run records status, timing, trace ID, optional outcome, error stack, and freeform summary metadata.

- Health is derived from recent runs at read time.

- Run detail exposes trace IDs for Convex log lookup.

- Manual Run now is available for configurable definitions, including disabled definitions, after a confirmation modal showing configured recipient rules.

- Config changes and manual Run now clicks are audited.

## Backend implementation

- Adds monitoringDefinitions for configurable report/reminder definitions.

- Adds monitoringRuns as the consolidated execution ledger.

- Adds schedule helpers for due-time checks and validation.

- Adds a generic monitoring scheduler cron that runs every 15 minutes.

- Wraps code-owned system jobs with the generic run recorder.

- Updates Buildout Freshness and REBL3 DD reconciliation to run from Monitoring definition config.

- Keeps public Monitoring functions capability-gated with admin.automations.read / admin.automations.manage.

## UI implementation

- Adds Monitoring home at /admin/monitoring-alerts.

- Adds detail route /admin/monitoring-alerts/[definitionKey].

- Shows configurable definitions separately from read-only system jobs.

- Provides schedule, recipient, enabled-state, run-history, trace, and Run now controls for configurable definitions.

- Preserves read-only run diagnostics for system jobs.

- Uses themed admin controls rather than native selects.

## Test plan

- [x] CI=1 pnpm --dir chat exec vitest run convex/automations/monitoring.test.ts convex/automations/monitoringDefinitions.test.ts convex/rhodesAccountabilityNotifications.test.ts convex/portfolio/dueDiligence.test.ts app/(main)/admin/monitoring-alerts/_components/monitoring-alerts-view.test.tsx app/(main)/admin/__tests__/admin-sidebar.test.tsx components/admin/__tests__/admin-nav-config.test.ts --reporter=default

- [x] node --test scripts/check-monitoring-cron-coverage.test.mjs

- [x] pnpm --dir chat typecheck

- [x] pnpm lint:read-bounds && pnpm lint:convex-paths && pnpm lint:boundaries

- [x] Targeted Biome check on touched files

## Summary

Adds a new Progress view to the Buildout dashboard (/dashboards?tab=buildout) for P2 members to track pre-launch Work Unit completion per campus.

Rows are non-cancelled sites whose current derived milestone is in the pre-launch buildout window (milestones 4-8: construction permits through ready-to-open). Columns are all 13 Quality Bars. Each populated cell is a percentage bar of pre-launch catalog WU completion for that QB; QBs with no pre-launch WUs for that site render as empty cells. Clicking a populated cell opens a side panel that drills into the QB -> its Work Unit Groups -> pre-launch Work Units, where eligible users can edit each WU's status, start/due/completed dates, and assignee. The panel also exposes read-only, paginated notes for WUs that have notes.

The existing FTO Pipeline matrix remains the default; the new grid is a second mode selectable from the "Sort & display" popover, mirroring how the Diligence tab switches between Progress and Outcomes. The only DB schema change is a new optional launchPhase ("pre" | "post") field on work units. Everything else the dashboard edits (title, status, dates, assignee) already exists on the workUnits table.

### Screenshots

<img width="2294" height="1220" alt="Screenshot 2026-07-06 at 1 49 28 PM" src="https://github.com/user-attachments/assets/6bbc3628-741c-4b04-a48f-94154213c903" />

<img width="2022" height="1212" alt="Screenshot 2026-07-06 at 1 49 56 PM" src="https://github.com/user-attachments/assets/4cac28e1-d415-4983-9690-15dc8906438c" />

<img width="968" height="478" alt="Screenshot 2026-07-06 at 12 02 43 PM" src="https://github.com/user-attachments/assets/0befac43-6aab-49d8-acb3-391e07b470bf" />

### Changes

Schema & data

- chat/convex/rhodes/schema.ts — adds launchPhaseValidator and an optional launchPhase field on the workUnits table.

- chat/convex/rhodes/p2WorkUnitDefinitions.ts — adds launchPhase: "pre" to the P2WorkUnitDefinition type and to all 60 P2 definitions (P2 has no post-launch WUs yet).

- chat/convex/rhodes/p2WorkUnits.ts — provisioning now stamps launchPhase on inserted WUs, and adds backfillWorkUnitLaunchPhase to set launchPhase: "pre" on existing P2 pre-launch catalog rows that lack it.

Backend queries & API

- chat/convex/rhodes/dashboard.ts — adds p2BuildoutDashboard (grid payload), p2BuildoutSiteQualityBarDetail (panel payload), p2BuildoutWorkUnitNotes (paginated WU notes), and editP2BuildoutWorkUnit. Reads are gated on operations.buildout.read; edits also require canEditSite. The bar numerator counts WUs whose status is completed or skipped; only pre-launch WUs count (missing launchPhase is treated as "pre" defensively). All 13 QB columns are emitted; per-row cells are only present for QBs with >=1 pre-launch WU.

- chat/lib/p2-buildout-contract.ts *(new)* — shared payload types + zod schemas for grid, detail, and notes payloads; shared edit types/status values. The edit request body schema lives in the API route.

- chat/lib/rhodes-p2-buildout-server.ts *(new)* — token-injecting fetch/edit helpers over the Convex functions, including the notes feed helper.

- chat/app/api/p2-buildout/route.ts *(new)* — GET grid payload.

- chat/app/api/p2-buildout/[siteId]/quality-bars/[qb]/route.ts *(new)* — GET QB detail.

- chat/app/api/p2-buildout/[siteId]/work-units/route.ts *(new)* — POST single-WU edit (status, dates, assignee).

- chat/app/api/p2-buildout/[siteId]/work-units/[workUnitId]/notes/route.ts *(new)* — GET paginated read-only WU notes.

Frontend

- chat/components/dashboards/buildout/buildout-view.tsx *(new)* — parent that holds the persisted Pipeline/Progress view mode and renders both panes (Pipeline = existing FtoPipelineView, Progress = new grid).

- chat/components/dashboards/buildout/buildout-view-mode-toggle.tsx *(new)* — the Pipeline/Progress segmented control, injected as the rootPrefix of the Sort & display popover.

- chat/components/dashboards/buildout/p2-buildout-progress-view.tsx *(new)* — fetches the grid, owns search/sort + panel selection state, and mounts the side panel.

- chat/components/dashboards/buildout/p2-buildout-matrix.tsx *(new)* — sticky Site column + one column per QB; populated cells show percentage bars, empty cells show no WU data.

- chat/components/dashboards/buildout/p2-buildout-detail-panel.tsx *(new)* — QB detail panel: header count, WUGs -> pre-launch WUs, per-WU editor (status, dates, assignee), compact navigation, close control, and read-only notes view.

- chat/components/dashboards/shared/note-card.tsx *(new)* — shared display component for note feed entries.

- chat/components/dashboards/dashboards-layout.tsx — the buildout tab now renders <BuildoutView /> instead of <FtoPipelineView />.

- chat/components/dashboards/fto/fto-pipeline-view.tsx — threads an optional, backward-compatible renderViewModeToggle prop so the toggle can be injected into the pipeline's Sort & display popover; existing behavior is unchanged.

Tests, docs, and related hardening

- chat/convex/rhodesDashboardP2Buildout.test.ts, chat/convex/p2WorkUnits.test.ts, chat/app/api/p2-buildout/__tests__/route.test.ts, chat/components/dashboards/buildout/__tests__/p2-buildout-matrix.test.tsx — coverage for dashboard semantics, backfill behavior, API route behavior, notes pagination, and matrix rendering.

- chat/lib/platform-error-coverage-inventory.ts and docs/platform-error-smoke.md — registers the new P2 Buildout routes in platform error coverage.

- chat/lib/operating-sites.ts and tests — accepts nullable split DRI refs returned by Rhodes listSites.

- chat/convex/rhodesDashboardParity.test.ts — tightens utilities parity assertions.

- features/dashboards/p2-buildout-dashboard/plan.md *(new)* — the implementation plan this PR was built from.

### Design Decisions

- Milestone-driven site selection — the Progress grid uses the same active-milestone model as the rest of the dashboard: sites are included when their current milestone is one of milestones 4-8, not merely because their stored stage is buildout or operating.

- All 13 QB columns are always visible — rows remain sparse, but the matrix shape is stable across campuses.

- launchPhase on WUs only — WUGs can hold both pre- and post-launch WUs, so the pre/post distinction lives on the WU. Adding the field to workUnits alone keeps the model minimal; the dashboard's pre-launch filter reads it directly.

- Status and completedDate are independent — toggling a WU's status never auto-stamps or clears completedDate (and vice versa). P2 members set the completion date explicitly.

- "Done" counts completed or skipped — matches the existing dashboard rollups so a legitimately skipped WU doesn't drag a campus's bar down.

- Missing launchPhase treated as "pre" — bars render correctly before the backfill runs, so the rollout is not gated on the migration.

## Backfill Runbook

This PR adds launchPhase to P2 work units. Existing P2 pre-launch catalog workUnits should be backfilled after deploy so rows missing launchPhase are explicitly patched with launchPhase: "pre".

The backfill is paginated by site. With scheduleNext: true, the command response reports only the current page's counts; later pages are scheduled and are not aggregated into the CLI output.

### Dev

First, confirm which dev deployment you are pointing at:

pnpm --dir chat exec convex dev --once

Then dry run against that dev deployment:

CONVEX_DEPLOY_KEY= pnpm --dir chat exec convex run rhodes/p2WorkUnits:backfillWorkUnitLaunchPhase '{"batchSize":5,"scheduleNext":true,"dryRun":true}'

Then run the live dev backfill:

CONVEX_DEPLOY_KEY= pnpm --dir chat exec convex run rhodes/p2WorkUnits:backfillWorkUnitLaunchPhase '{"batchSize":5,"scheduleNext":true,"dryRun":false}'

### Production

Dry run production first:

CONVEX_DEPLOY_KEY= pnpm --dir chat exec convex run --deployment oceanic-pika-463 rhodes/p2WorkUnits:backfillWorkUnitLaunchPhase '{"batchSize":5,"scheduleNext":true,"dryRun":true}'

Then run the live production backfill:

CONVEX_DEPLOY_KEY= pnpm --dir chat exec convex run --deployment oceanic-pika-463 rhodes/p2WorkUnits:backfillWorkUnitLaunchPhase '{"batchSize":5,"scheduleNext":true,"dryRun":false}'

Expected result: existing P2 pre-launch catalog workUnits missing launchPhase are patched with launchPhase: "pre". Unrelated/manual WUs are left alone, and existing launchPhase: "post" rows are preserved. The dashboard has a defensive fallback before this runs, but the backfill should still be part of the PR rollout.

## Test Plan

- [x] pnpm typecheck — clean (exit 0)

- [x] pnpm biome check — clean

- [ ] Manual: run rhodes/p2WorkUnits:backfillWorkUnitLaunchPhase with {"dryRun": true}, review counts, then {"dryRun": false} to materialize launchPhase: "pre" on existing WUs

- [ ] Manual: confirm operations.buildout.read is granted to the roles that should see the Progress grid

- [ ] Manual: on /dashboards?tab=buildout, switch to Progress, verify all 13 QB columns, per-QB bars/empty cells, side-panel navigation, WU notes, and WU edits for status/dates/assignee

## Summary

Renames the Redshift tables for the netsuite-gl-detail and netsuite-unrealized-gains pipelines to a month_end_ prefix so they're easy to recognize when scanning tables, and refreshes each pipeline's display name/description in the Surtr app.

| | Old | New |

|---|---|---|

| Raw (gl-detail) | staging_netsuite.gl_detail | staging_netsuite.month_end_gl_detail |

| Raw (unrealized) | staging_netsuite.unrealized_gains_losses | staging_netsuite.month_end_unrealized_gains_losses |

| Shared enriched | core_finance.fct_pi_gl_enriched | core_finance.month_end_fct_pi_gl_enriched |

| source_table tags | gl_detail / unrealized_gains_losses | month_end_gl_detail / month_end_unrealized_gains_losses |

Display metadata:

- pipeline_name → NetSuite Month End GL Detail Pipeline / NetSuite Month End Unrealized Gains/Losses Pipeline

- description now names the new staging + enriched tables (within the 100/500-char registry limits)

## Changes

- pipeline.json (both): pipeline_name, description, REDSHIFT_TABLE

- src/redshift_loader.py defaults, src/enrichment.py ENRICHED_TABLE + SOURCE_TABLE_TAG, src/handler.py docstrings

- Test assertions for the new source_table tags

- READMEs + docs/features/00 & 04

- run_local.py: new --periods flag (local dev only) to backfill an explicit list of periods through the full handler path

- unrealized-gains/uv.lock: pin python-dotenv as a main dependency (matches gl-detail)

## Data migration / rollout

- Historical data was backfilled locally into the new tables (2021–2025) via run_local.py --periods, running the full production path (NetSuite → S3 → Redshift + AI enrichment).

- The old tables are intentionally left in place (frozen) so downstream consumers (e.g. Klair Schedule C1/C2) keep working. They will be migrated to the new tables after deploy, then the old tables dropped manually. No ALTER RENAME / in-place migration.

## Verification

- ruff check + ruff format --check: clean (both pipelines)

- pytest: gl-detail 68 passed, unrealized-gains 93 passed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Automated fix for tesorio-collections-sync — fix_class code_fix.

Resolves https://github.com/AI-Builder-Team/Surtr/issues/618

- Run: a72a9b97-7486-45d0-9f0f-b260433d1ca2

- Signature: 1b63b2b053d308c5c10e84d733f5535d3506fec82af0d16e9f2fe3513ff7c66e

- Tests: green

---

🤖 Opened by the triage agent. A human must review before merge — the

agent's diff is confined to the pipeline directory and is not auto-merged.

## Summary

Gives the Book Value → Schedule A Total row a per-transaction GL drill-down with "Open in NetSuite" links, and extends the same detail to the Book Value Report's "Other Investments" row (which shares Schedule A's exact filter).

Previously Schedule A's Total drilled into an account-level aggregation (GroupedDetailPanel, no transaction identity / links), even though its individual entity rows — and Schedule B/C1's Totals — already showed GL-line rows with NetSuite links. This closes that asymmetry and mirrors the recent "Net Flows to/from Loan Book" ↔ Schedule B change.

## Changes

Backend (klair-api)

- book_value_schedules_service.py: add fetch_schedule_a_total_gl_detail(period) — GL-line rows grouped by entity_name using the identical Schedule A filter (source_table='gl_detail', account_name ILIKE '%Other Long%Term%', is_cash_transaction=TRUE, amount_net>0, is_reversal=FALSE), selecting document_number + internal_id (SPLIT_PART(line_id,'-',1)) for the links.

- finance_monthly_financial_reporting_router.py: add GET /schedule-a-total-gl-detail.

- Tests: TestFetchScheduleATotalGLDetail (filter/YTD binding + grouping/sign) and add the new fetcher to the parametrized "every GL query selects internal_id" guard.

Frontend (klair-client)

- monthlyFinancialApi.ts: add fetchScheduleATotalGLDetail.

- BookValueView.tsx: Schedule A Total row now renders GroupedGLDetailPanel; route the Report Report/Alt "Other Investments" row and the Bridge tab's Current/Prior columns to the same GL-line detail via a shared renderOtherInvestmentsGLDetail helper. Dropped the now-unused fetchScheduleATotalDetail import.

## Behavior notes

- Drill-downs show raw GL amounts grouped by entity (consistent with Schedule B; ties to the Schedule A summary total).

- Schedule A is cash-only / positive-only by design, so non-cash journal reclasses stay excluded — a NetSuite reclass concern, not a display one.

- Bridge tab: the Change column keeps its Current-vs-Prior netting view (per-transaction rows don't map to a delta); the Current/Prior columns get the NetSuite links.

## Testing

- Backend: ruff format/check clean, pyright 0 errors, 62/62 book-value service tests pass.

- Frontend: tsc --noEmit clean, eslint --max-warnings 0 clean, 174/174 MFR detail-panel tests pass.

- Not yet driven in a live browser (needs an authenticated dev session); reuses the same panel/endpoint pattern as Schedule B/C1 Totals.

🤖 Generated with [Claude Code](https://claude.com/claude-code)