The Trilogy Times

## Summary

- add an isolated AWS Fargate implementation path with a trusted controller, credential-free executor, and fixed-repository private clone broker

- reuse Ezio's real model loop, validation profiles, candidate format, and draft-only publisher

- require ECS STOPPED state before a separate trusted publisher can mint a GitHub write token

- persist CloudWatch telemetry and versioned S3 diagnostics in s3://ezio-diagnostics/dev/<run-id>/, with prod/ reserved

- remove the superseded Cloudflare POC after dry-run and draft-publication parity; no webhook is live until AWS ingress is added

## Security boundaries

- executor has no task role, secrets, repository credentials, or public egress during model execution

- repository scope remains fixed in deployment configuration

- clone broker permits only Git smart-HTTP read operations for allowlisted repositories

- controller stops the executor before Step Functions records the publication gate

- publisher independently rechecks the stopped task and always creates draft PRs only

- trusted candidate capture retains existing cache/artifact rejection rules

## Deployed evidence

- fargate-proof-shared-diagnostics-v8-20260804 passed private execution, credential isolation, checkpoint restore, forced stop, egress denial, and stopped-before-publication gating

- fargate-klair-3452-dry-v4 ran the real model loop and passed validation with one exact candidate file

- fargate-klair-3452-publish-v1 passed the independent gate and created draft Klair PR #3470

- the development stack is ezio-fargate-dev-proof in us-east-1; it remains deployed and may incur costs

## Cloudflare teardown

- deleted Workers ezio and ezio-dev, all four Ezio Workflows and their instances, both reconciliation queues, both sandbox container applications, Worker secrets/bindings, and Durable Object classes/data

- purged 4,199 objects from ezio-dev-run-diagnostics and 6,002 objects from ezio-run-diagnostics, then deleted both R2 buckets

- verified that no Ezio-named Workers, Workflows, queues, containers, or R2 buckets remain; unrelated account resources were untouched

## Validation

- 217 Vitest tests passed

- application and runtime TypeScript checks passed

- both Wrangler dry-run Worker builds passed

- Ruff formatting and checks passed on modified Python files

- modified TypeScript and Markdown formatting checks passed

- shell syntax checks and CloudFormation validation passed

- Fargate v8 boundary proof passed against s3://ezio-diagnostics/dev/

## Remaining migration gate

There is currently no live GitHub webhook. Authenticated AWS ingress, broader reconciliation parity, and operating-cost comparison remain follow-up migration gates.

Linear: [AERIE-797](https://linear.app/builder-team/issue/AERIE-797)

## Summary

- Delivers AERIE-797 — Multi-file skills: a user-authored skill is no longer a single Markdown string but a versioned bundle of a required SKILL.md plus optional textual resources (references, examples, templates, config, read-only scripts). The change spans the full stack — shared contracts, Convex persistence/lifecycle, filesystem projection, the sync reconciler, the Durable Flue worker, and the Context editor UI — behind one canonical bundle definition.

- Canonical bundle contract (@bran/contracts): one authoritative multi-file model with safe/normalized POSIX paths, per-file and whole-bundle size/count limits, a whole-tree content hash, and a legacy-scalar compatibility shim so existing single-file skills keep working with no migration.

- Convex persistence & lifecycle: skill versions store a complete immutable file set. Create / Save / Replace (whole-bundle) / Archive / Restore / version navigation / change summaries / previous-version Diff all operate on complete bundles. Uploads stream through the existing bounded upload transport (begin → declare → chunk → finalize) rather than an inline mutation envelope.

- File-anchored comments: comments are anchored to an immutable (version, file path) and survive independently of edits, extending the existing comment model to every text file in a bundle.

- Filesystem projection + sync: the complete manifest is projected through a write-boundaried, bounded-HTTP projection path; the sync reconciler rebuilds the exact committed bundle, with Archive/Restore reconciliation and stale-event safety.

- Durable Flue runtime: the run-pinned bundle is served over the executor-authenticated /agent/runs/skills endpoint and registered with the worker via defineSkill({ files }). Resources load lazily through read_skill_resource; the run is pinned to the exact immutable version selected at run creation. The resource protocol is additive for mixed-deploy rollout, integrity is validated fail-closed per skill, and script files are read-only resources (execution is out of scope).

- Context editor UI: the skill surface gains a File Tree (browse/select; structure changes only via Replace), open-file tabs, dedicated Name/Description fields, a language-aware body editor, a multi-file version Diff (changed-path list + per-file diff), and Create/Upload/Replace review-before-persist flows (single SKILL.md, .zip, or folder). Flat skills and prompts keep their existing single-file experience.

- Rollout safety: CI adds a projection-volume preflight and rollback ordering so the projection volume is provisioned/verified before dependent deploys.

## Sub-issues included

| Issue | Slice |

|---|---|

| [AERIE-797](https://linear.app/builder-team/issue/AERIE-797) | Parent — multi-file skills with File Tree UI for Durable Flue agents |

| [AERIE-798](https://linear.app/builder-team/issue/AERIE-798) | Canonical multi-file bundle contract, persistence, hashing, legacy compatibility |

| [AERIE-801](https://linear.app/builder-team/issue/AERIE-801) | Bundle lifecycle: version nav, Create/Save/Replace/export, Archive/Restore |

| [AERIE-802](https://linear.app/builder-team/issue/AERIE-802) | Complete-manifest filesystem projection + reconciliation + stale-event safety |

| [AERIE-799](https://linear.app/builder-team/issue/AERIE-799) | Run-pinned resource protocol + native lazy Flue resource registration |

| [AERIE-800](https://linear.app/builder-team/issue/AERIE-800) | File-aware, immutable-version skill comments |

| AERIE-797 (UI slices 06–10) | Context UI foundation, multi-file editor, flat editor, Create/Upload/Replace, multi-file Diff |

## Why

Skills were modeled as one Markdown string, a single-file assumption baked into Convex storage, versioning, comments, projection, run snapshots, and the Durable Flue worker. Operators could not author skills that rely on companion files, and the Convex/user-authoring path could not express the folder-based bundle the Flue runtime already understands. This closes that gap with one canonical bundle definition shared across every layer, without changing the Flue/CD topology or conflating readable resources with executable code.

## Business Value

Operators can build richer skills — instructions plus reference docs, examples, and templates — without an application deployment, keep agent instructions concise while detailed collateral loads lazily on demand, round-trip skills between local authoring and Aerie via upload/download, and get versioning, comments, replacement, and runtime delivery all operating on one coherent bundle.

## Backward compatibility

- No migration required. Existing single-file skills continue to work; the legacy-scalar shape is normalized into the canonical bundle and served to old/new workers via the additive protocol (new worker synthesizes legacy behavior when resources are absent; old worker ignores the extra fields).

- No destructive schema changes; existing permission boundaries are preserved and backend/runtime validation is added.

## Test plan

- [x] chat + aerie-flue-agent-worker typecheck (tsc --noEmit)

- [x] Biome check on changed files

- [x] Unit/integration suites: contracts (bundle/transport/projection/comment validation), Convex (lifecycle, comments, upload transport, run-pinned endpoint, projection boundary), sync reconcile, Flue worker (bundle validation + lazy resource registration + integrity/compatibility), and Context UI (file tree, editor, upload/replace, multi-file diff, comments)

- [x] Manual QC of Create/Upload/Replace, multi-file Diff, and file-anchored comments against the validated mocks

- [x] Cloud smoke: activate a multi-file skill and lazily read a nested resource through a Durable Flue run

## Known follow-ups (explicit non-goals)

- Executing uploaded scripts — .py/.sh are stored and hydrated as read-only text; sandboxed execution is a separate feature.

- Multi-file download/export — flat skills export SKILL.md; the full-bundle (ZIP) export path is a follow-up slice.

- In-editor structural CRUD — add/remove/rename/move are performed by uploading a corrected bundle through Replace.

- Bulk import, arbitrary version-to-version comparison, historical-version restore, unsaved-draft test runs, concurrent-editing redesign, and binary/large-asset storage remain out of scope.

- Frontend decomposition into per-view Linear sub-issues is intentionally deferred.

🤖 Generated with [pi](https://pi.dev)

Follow-up to #1109, which is already merged and live. Three changes, all driven by one correction: the audience is internal and token-gated, not external.

## Why this exists

The scope note I built #1109 to said *"Because the audience is EXTERNAL app customers … do NOT expose internal infrastructure identifiers."* That premise is withdrawn. These endpoints are consumed internally behind a pak_ token.

That inverts the right answer. Withholding detail from an internal consumer diagnosing "can I trust this data?" only makes the answer unactionable — and every consumer here already has the AWS and Redshift access to see all of it directly. The redaction was defending against nothing while degrading the primary use case.

It was also actively harmful. In production its 12-digit rule fired on a Google Sheets quota error and turned project_number:493… into project_number:[redacted] — a legitimate diagnostic value, corrupted.

## 1. Redaction removed

The [redacted] pattern list is deleted, not disabled. Error text now ships whole — traceback, Where: SQL context, warehouse object names — capped at 4000 chars instead of 500.

The orchestration envelope is still unwrapped, but for signal, not secrecy: a Step Functions task failure is States.TaskFailed: {several KB of ECS task state} with no failure reason anywhere in it. The prefix plus the run's log group is the useful answer, and the log group now ships.

- "error": "Task [redacted] exited on [redacted]"
+ "error": "Task arn:aws:ecs:us-east-1:…:task/abc123 exited on 10.0.4.17"

## 2. Withheld fields restored

| Restored | Why it matters internally |

| --- | --- |

| infra.step_function_arn / step_function_url | jump straight to the execution |

| infra.cloudwatch_log_group / cloudwatch_logs_url | the fastest path from "failed" to the cause |

| per-run logs.cloudwatch_log_stream | the exact stream for that run |

| owners (name + email) | who to page |

| deployed_at | when the pipeline last changed |

| output_summary | the run's own result line (18,442 rows written to …) |

| triggered_by | who ran it |

| observer.model_id / observer_version | the first things you want when a verdict looks wrong |

The original goal asked for the infra links; I only dropped them because of the "external" note.

Still not served: run input_params. They can carry caller-supplied values unrelated to the run's health, and nobody has asked for them.

## 3. A real bug production exposed — open_finding_count contradicted verdict

verdict came from the latest observation; open_finding_count and worst_severity spanned the 7-day window. So 42 of 90 live pipelines returned this, which reads as a contradiction:

{ "verdict": "OK", "open_finding_count": 8, "worst_severity": "H" }

The 8 were from earlier runs that had since gone green. The counts beside verdict are now scoped to the same evaluation, and the window totals keep their own names:

{
"verdict": "OK",
"open_finding_count": 0, "worst_severity": null,
"window_finding_count": 8, "window_worst_severity": "H",
"window_days": 7
}

observer.findings[] on the detail endpoint is unchanged — still the full window view with occurrences and first_fired_at / last_fired_at, which is what it's for.

This one is audience-independent; it was wrong either way.

## Tests

test/pipeline-api/ — the presentError suite was rewritten around the new contract rather than deleted:

- the traceback and Where: SQL context now survive (previously asserted truncated)

- an ordinary sentence containing an ARN and a private IP passes through untouched

- a Step Functions blob still collapses to its prefix — asserted as "holds no reason", not "leaks topology"

- cap is 4000, and a 3999-char message is untouched

- the leak-scan test became "serves the operational detail an internal consumer needs to act" — asserts owners, infra, per-run logs, output_summary, triggered_by, model_id, observer_version are all present, and input_params is not

- new test pins the finding-count fix on both endpoints: latest-run clean after a bad week → verdict: OK, open_finding_count: 0, window_finding_count: 1

## Gates

| Gate | Result |

| --- | --- |

| pnpm lint (biome) | pass — 95 files |

| pnpm build (tsc, CI-equivalent) | pass — 0 errors in src/ or test/ |

| pnpm test:unit (what CI runs) | pass — 45 files |

| targeted tsc on the touched app/ files | pass |

I did not run pnpm build:ui: a dev server is running from this worktree for the owner, and next build shares .next with it — I broke that process once already this way. The app/ diff is six string literals (app customers → internal consumers, tab-name fix), the jsdom UI tests pass, and tsc is clean on those files.

## Note on #1109

#1109 was squash-merged while I was working. My push recreated the deleted feat/pipeline-status-api branch by accident; I've moved the commit onto this fresh branch off main and deleted the resurrected one. No commits lost, main untouched.

## Follow-up commit: envelope detection was too loose

Caught while dumping the restored payload shape, before this was reviewed. unwrapFailure searched for the first { or [ anywhere in the message, so any error text containing one was treated as a wrapped payload and collapsed to whatever preceded it. The casualty was the exact string that exposed the redaction problem:

in    APIError: [429]: Quota exceeded for quota metric 'Read requests' … for consumer 'project_number:493812345678'.
was   APIError
now   (unchanged, in full)

Same for the Python reprs the runners emit inline — ProgrammingError: {'S': 'ERROR', …} — which lost the staleness detail in front of them.

Detection is now { immediately followed by a double-quoted key. Both real envelope shapes match; single-quoted dicts and bracketed status codes are prose and stay. Unrecognized input is returned untouched rather than replaced with a placeholder (the OMITTED_DETAIL string is gone).

Verified against all six failure shapes present in prod; the two real strings are regression tests.

## Fields restored — full audit

| Field | Status |

| --- | --- |

| infra.step_function_arn, infra.step_function_url | restored |

| infra.cloudwatch_log_group, infra.cloudwatch_logs_url | restored |

| per-run logs.cloudwatch_log_stream (+ group, URLs, SFN URL) | restored |

| owners (name + email) | restored |

| deployed_at | restored |

| output_summary | restored |

| triggered_by | restored |

| observer.model_id, observer.observer_version | restored — see note |

| run input_params | still withheld |

| braintrust_span_id | still withheld |

model_id / observer_version are arguably noise. I restored them because they are the first two things you want when the verdict itself looks wrong — a stale observer_version explains a verdict produced under an older rubric. Two scalars, easy to drop if you disagree.

input_params and braintrust_span_id stay out: the former can carry caller-supplied values unrelated to health, the latter is an internal eval-tooling handle with no meaning to a consumer.

## Not released

No main→production PR, no merge to production, no deploy, no announcement. Until you release, prod serves the old redacted/stripped shape — production is at 1941b784 (release #1111), which carries #1109 only.

Worth verifying the full error text against real prod data once this does ship: that's where the redaction was doing visible damage, and it's also where the envelope-detection bug above would have shown up.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Problem

Book Value → Other Investments read \$118,817,795 YTD Jul-2026, missing a \$19,084,140.75 wire — [NetSuite journal 48579698](https://4914352.app.netsuite.com/app/accounting/transactions/journal.nl?id=48579698) (Jun 2026, account 25500). Raised by a stakeholder as a missing transaction.

The amount is in the warehouse. It was filtered out because is_cash_transaction = FALSE, and Klair's query gates on is_cash_transaction = TRUE.

## Root cause

The flag is judged by Claude Haiku from this prompt rule:

> is_cash_transaction: true if this is a wire transfer (memo starts with "Sending Bank:").

The model generalises *starts with* → *contains* for most rows — 105 of 106 historical ticket-prefixed wires (FINANCE-145762: Sending Bank: ...) were correctly flagged TRUE. But when the same wire evidence is wrapped in journal-reclass wording, it reads as a reclassification:

FINTXN-57236 | GJ Request - Jan - 05312026 - June reclasses v3 - Sending Bank: 111000753, COMERICA TEXAS, ...

Replaying the production prompt at temperature=0: False 6/6 for this row; control rows True 3/3.

An audit found the same memo families judged both ways:

| Memo family | True | False |

|---|---|---|

| WAVESYSTEMCO (TPA ACH) | 13 | 2 |

| TPA FUNDS TRANSFER | 1 | 1 |

| YOUR REF= / PAID TO= / REC FROM= | 4 | 11 |

Introduced 18 Mar 2026 (Klair #2251), when a working deterministic str.startswith() check (#2182, 12 Mar) was converted into prompt text. Exposure window ~4.5 months.

## Fix

Assert the flag from objective memo markers, unioned with the model's answer:

"is_cash_transaction": bool(parsed.get("is_cash_transaction", False)) or _has_cash_marker(memo),

The union matters: 20 rows have no listed marker but real wire vocabulary the model recognises (e.g. a bare correspondent-bank address). Those keep working. This mirrors the existing deterministic override for account 22916 loan rows in the same function.

## Validation — all 3,108 production rows

| | |

|---|---|

| Rows flipped False → True | 16 |

| Entering Book Value Other Investments | 7 (\$34,742,860.75) |

| FY2026 | 1 (\$19,084,140.75) |

| Mark-to-market / TB rows swept in | 0 ✅ |

YTD Other Investments: \$118,817,795 → \$137,901,936.

⚠️ Historical figures (2022–2024) only move if someone runs a backfill — those periods are outside the rolling window, so deploying alone is inert for closed years. Needs Finance sign-off before any backfill.

## Tests

TestCashTransactionMarkers — 10 tests from real production memos, each watched failing first. Mutation-checked three ways (removing the union → 7 fail; contains→startswith → 6 fail; dropping case-insensitivity → 3 fail). Guardrails pin mark-to-market and TB-upload memos to False.

83/83 pass (was 73). ruff check + format --check clean on the CI-pinned 0.15.22.

## Memo edit was tried first and ruled out

The stakeholder updated the NetSuite memo and the pipeline was re-run on demand — the row stayed False. Three rewrites were tested against the live prompt:

| Variant | Result |

|---|---|

| As edited by stakeholder | False 3/3 |

| Lead with Sending Bank:, keep reclass wording | True 3/8 ⚠️ unstable |

| Lead with Sending Bank:, drop reclass wording | True 8/8 |

The only reliable variant requires deleting the *accurate* "GJ Request / June reclasses v3" wording. And the middle row returns different answers on identical input — so even a passing memo could silently regress, since every row is re-judged daily.

## Not covered by this fix

13 rows typed Deposit in NetSuite (a cash receipt by definition) are flagged False; 6 reach Other Investments (~\$7.17M). Twelve carry no wire vocabulary at all (Comerica ID 35186, Payment to Hudson Bend Purchase), so no memo-parsing rule can catch them. A type = 'Deposit' rule is the likely fix — pending Finance confirmation that a Deposit on these accounts always means cash.

## Deploy

Merge → deploy → re-run netsuite-gl-detail for Jun 2026 (still in the rolling window until 31 Aug). Verify:

SELECT line_id, is_cash_transaction, _loaded_at
FROM core_finance.month_end_fct_pi_gl_enriched WHERE line_id = '48579698-0';

Klair has a 60s TTL cache on Book Value — allow a minute before refreshing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Fixes [KLAIR-3115](https://linear.app/builder-team/issue/KLAIR-3115/qtd-reports-gate-cost-action-items-on-dollar25k5pp-materiality)

## Problem

The July 2026 AI Engineering & Builder QTD doc surfaced this action item:

> P2 — HC Expenses is over budget (100.3% used vs 100% budget)

> Review HC Expenses spend immediately. Actual $240,718 exceeds budget $239,917 by $802.

An $802 overrun on a $239,917 plan is a 0.33% variance — not an SVP-actionable signal. Raised by stakeholder review, who correctly expected a materiality cap to suppress it.

Every number rendered is correct. This isn't a data or arithmetic defect — it's a missing gate in the rule that decides *whether to speak at all*.

## Root cause

The cost-overspend rule in metrics.py fired on four conditions, where status == "over_budget" is set purely by pct_used > 100.0. No dollar or percentage floor — so $1 over budget on any non-internal cost line emitted an action item.

Dollars entered only *after* the rule fired, to pick severity:

priority = "P1" if overage >= _P1_DOLLAR_OVERAGE else "P2"   # $50K

_P1_DOLLAR_OVERAGE is a severity split, not a suppression gate — nothing sorts below P2, so everything surfaced. That's likely the source of the "we already had a cap" recollection: $50K is right there in the rule and reads like a threshold, but it never silenced anything.

## The change

_MATERIAL_OVERAGE_DOLLARS = 25_000.0
_MATERIAL_OVERAGE_PP = 5.0
is_material_overage = (
overage >= _MATERIAL_OVERAGE_DOLLARS or pct_off >= _MATERIAL_OVERAGE_PP
)

Added as a fifth condition. _P1_DOLLAR_OVERAGE untouched.

### Why OR, not AND

Measured across all 27 units in the July 2026 run — 54 cost lines currently fire:

| Gate | Kept | Suppressed |

|---|---|---|

| Today (none) | 54 | 0 |

| Dollar-only (>= $25K) | 36 | 18 |

| Combined ($25K OR 5pp) | 50 | 4 |

A dollar-only cut would have silenced 14 legitimate signals — small-budget lines running multiples over plan (e.g. COO Service NHC OPEX: $935 over a $500 plan = 187pp). The OR keeps them.

Suppressed (all genuine noise): GT HC Expenses ($466 / 0.09pp), AI Eng & Builder HC Expenses ($802 / 0.33pp — the reported case), Zax NHC OPEX ($1,381 / 1.86pp), Skyvera NHC COGS ($18,661 / 3.44pp).

## Verification

- 6 new unit tests — both legs alone, both-below, and both boundaries at exactly threshold

- Mutation-checked in both directions, independently by implementer and reviewer: with the gate zeroed the suppression tests fail; with it set very high the "still fires" tests fail. The tests genuinely detect their target.

- Full suite: 779 passed, 7 deselected. ruff + pyright clean.

- End-to-end preview docs generated (no ledger writes) confirming both directions in rendered output:

- AI Eng & Builder → "No action items identified."

- COO Service → P2 NHC OPEX item still present at 287% of budget

- Blast radius: zero. Two non-test callers; the empty-list path already renders "No action items identified." and is covered by existing tests. No API, MCP, email, or frontend consumer.

## Reviewer notes

On the doc_builder relationship. The threshold *values* are borrowed from _VARIANCE_DOLLAR_THRESHOLD / _VARIANCE_PCT_THRESHOLD, but the *combination* deliberately differs: this gate is OR, _color_for_line is AND. They do not agree, by design — a line can render uncolored ("on plan") and still emit an action item when only one leg is material. That divergence is the point; it's what preserves the 14 small-dollar/high-percentage signals. An earlier draft of this PR claimed the opposite in its comments; that was wrong and is corrected in 4d3020eba / efb5e9382. Please don't "fix" the or to and for consistency — that would silence exactly the signals this exists to keep.

Commentary grain. The AI Eng & Builder doc now has no action items while its Executive Commentary still discusses HC Expenses — the gate works on the total line ($801 / 0.33pp, immaterial), the commentary on team rooms (AI.Eng.Engineer +$31,577, AI.Eng.Renewals +$12,784, netting against AI.Eng.Builder −$41,760). Both correct at their own grain; the doc reads coherently. If Finance wants team-room overages to raise action items, that's a separate change.

Unrelated file in the diff. klair-client/.../actionHubFilter.vitest.ts has a 3-line Prettier reformat, swept in automatically by a pre-commit hook during the main merge. Formatting only, no logic. Happy to strip it if preferred.

## Out of scope

- HC-underspend rule — same latent defect (bare pct_used < 70.0, no dollar floor). Deliberately deferred per stakeholder decision; worth its own ticket.

- Revenue rules — already gated by _AMBER_REVENUE_DOLLAR_THRESHOLD.

- Backfill — not required. Shipped July docs stay as-is; the fix applies from the next generation cycle.

## Deployment

⚠️ Scheduled cron runs will not pick this up until the klair/scheduled-jobs ECS image is rebuilt — that's a manual docker build -f Dockerfile.jobs (no CI). The rebuild would also pull in the deferred Education feature (#3417).

Threshold tuning: if Finance wants Skyvera NHC COGS ($18,661 / 3.44pp) surfaced, lower the dollar leg to $15K — one line.

## Testing

- This has been verified live with Ravi and all is looking good.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Summary

- add an isolated AWS Fargate implementation path with a trusted controller, credential-free executor, and fixed-repository private clone broker

- reuse Ezio's real model loop, validation profiles, candidate format, and draft-only publisher

- require ECS STOPPED state before a separate trusted publisher can mint a GitHub write token

- persist CloudWatch telemetry and versioned S3 diagnostics in s3://ezio-diagnostics/dev/<run-id>/, with prod/ reserved

- remove the superseded Cloudflare POC after dry-run and draft-publication parity; no webhook is live until AWS ingress is added

## Security boundaries

- executor has no task role, secrets, repository credentials, or public egress during model execution

- repository scope remains fixed in deployment configuration

- clone broker permits only Git smart-HTTP read operations for allowlisted repositories

- controller stops the executor before Step Functions records the publication gate

- publisher independently rechecks the stopped task and always creates draft PRs only

- trusted candidate capture retains existing cache/artifact rejection rules

## Deployed evidence

- fargate-proof-shared-diagnostics-v8-20260804 passed private execution, credential isolation, checkpoint restore, forced stop, egress denial, and stopped-before-publication gating

- fargate-klair-3452-dry-v4 ran the real model loop and passed validation with one exact candidate file

- fargate-klair-3452-publish-v1 passed the independent gate and created draft Klair PR #3470

- the development stack is ezio-fargate-dev-proof in us-east-1; it remains deployed and may incur costs

## Cloudflare teardown

- deleted Workers ezio and ezio-dev, all four Ezio Workflows and their instances, both reconciliation queues, both sandbox container applications, Worker secrets/bindings, and Durable Object classes/data

- purged 4,199 objects from ezio-dev-run-diagnostics and 6,002 objects from ezio-run-diagnostics, then deleted both R2 buckets

- verified that no Ezio-named Workers, Workflows, queues, containers, or R2 buckets remain; unrelated account resources were untouched

## Validation

- 217 Vitest tests passed

- application and runtime TypeScript checks passed

- both Wrangler dry-run Worker builds passed

- Ruff formatting and checks passed on modified Python files

- modified TypeScript and Markdown formatting checks passed

- shell syntax checks and CloudFormation validation passed

- Fargate v8 boundary proof passed against s3://ezio-diagnostics/dev/

## Remaining migration gate

There is currently no live GitHub webhook. Authenticated AWS ingress, broader reconciliation parity, and operating-cost comparison remain follow-up migration gates.

<!-- CURSOR_AGENT_PR_BODY_BEGIN -->

## Summary

The addresser can fix a finding, document it, then report it as unreported and tell the operator to "re-fire the addresser" — which is actively harmful (redoes finished work, posts duplicate replies, adds a second writer on the branch). Fixes reconciliation in crossCheckAccountability on a stable finding identity (path + body content, not a dimension label) plus a round-diff consultation against the tree, so a genuinely-completed fix is never reported as missing. Adds a new terminal already-fixed action distinct from both fixed and unreported, and replaces the "re-fire the addresser" remediation advice with the shipped, safe --unaddressed-only path.

## Did AI-133 regress, or is this a second route? (first deliverable)

AI-133 did not regress. Its wildcard-alias reconciliation (findingLookupKeys / crossCheckAccountability pass 2) still passes byte-identical, and its mechanism only ever reconciles an expected finding against an ADDRESS_REPORT bullet that exists but carries a mismatched or wildcard dimension label (e.g. Mercy's [unknown · ] vs. the agent inventing [Critical · unknown]).

The PR #141 (src/receipt-s3.ts:632) and PR #138 (two round-1 findings, f91cebc/4b82d04) cases are a different failure mode: the agent emitted zero bullets referencing the finding at all. There's nothing for any dimension-matching scheme — however wildcarded — to bind to, because there's no reported entry in the first place. That's a second, previously-uncovered route to the same wrong answer, not a regression of AI-133's fix.

## Why It's Needed

- unreported is the metric operators use to judge whether the addresser drops findings — fixed-but-uncredited findings inflate it, which would mislead AI-213's reviewer-precision baseline and any escape-rate analytics built on it.

- Confirming the #141 case required reading the branch by hand — doesn't scale to an unattended batch.

- The prior "re-fire the addresser" advice is one of two false-alarm classes found in a single session (AI-295 is the other) — the harness misreporting its own success is the costlier failure mode for unattended operation: a real problem stops one PR, a false alarm stops a PR *and* erodes trust in the gate.

## Changes

- crossCheckAccountability gains a pass 3 — identity-based fallback (findingIdentityKey / findingBodyHash, sha256-truncated) that anchors on the finding's path appearing in a not-yet-claimed reported entry's text plus a body/rationale overlap (bodyMatchDistance). Catches bullets whose header never parses into [sev · dim] path:line at all, so reconciliation no longer depends on a dimension token existing on either side.

- New optional RoundDiffEvidence (path → touched new-side line ranges) built from a gh compare between the review's commit_id (new field on PostedReview, populated from GitHub's review commit_id) and the current PR head, via the new fetchRoundDiffEvidenceViaGh / opt-in AddressInput.fetchRoundDiffEvidence hook. When a finding has zero report evidence but its cited location demonstrably changed in this diff, it reclassifies to the new already-fixed terminal instead of a flat unreported. Comparing against the review's *original* commit (not just one round's own before/after) recovers PR #138's cross-round shape too, where the fixing commits landed in a later round than the one whose thread went stale.

- already-fixed is added to ADDRESSER_ADJUDICATION_ACTION_RE (terminal — resolves the thread, excluded from future --unaddressed-only re-scoping) and excluded from unaddressedHighSeverity, but kept out of unreported so it never inflates that accountability count. Surfaced in AddressOutcome.alreadyFixed, the address_completed event (alreadyFixedFindings, append-only optional), renderAddressOutcome's tally line, and the next round's re-review prompt (buildPriorRoundBundle).

- The unreported reply's advice no longer recommends re-firing; it names drones address -p <PR> --review-id <ID> --unaddressed-only.

- The diff-evidence hook is opt-in, not defaulted inside addressFindings (unlike fetchPostedReview/postFindingReplies) — its absence is a safe no-op, so every existing test is unaffected. Wired explicitly at all three production call sites: runner.ts auto-fire, cli/address.ts standalone, and Mercy watcher.

- docs/decisions.md: appended a ledger row per repo convention (append-only).

Contract surface affected: AddressedFinding["action"] and PriorRoundDisposition["action"] gain the "already-fixed" literal (additive, all consumers checked — only renderReplyBody's switch needed a new case since it's the one place that branches on every literal). AddressOutcome's "completed" variant gains a required alreadyFixed: AddressedFinding[] field; both construction sites updated, and downstream readers (renderAddressOutcome, buildPriorRoundBundle) defensively fall back to ?? []/?.length ?? 0 for pre-existing hand-built test fixtures that predate this field (test files are outside tsc's scope per tsconfig.json, so this isn't just belt-and-braces).

## Breaking Changes

None. All new fields are additive/optional; the diff-consultation mechanism is strictly opt-in and never fires unless a caller wires fetchRoundDiffEvidence and the review has a resolvable commitId.

## Test Plan

- npx vitest run src/addresser.test.ts → 163 passed (was 137; +26 new tests covering the #141 and #138 regression fixtures, the identity-fallback pass with/without over-match guards, diff-evidence helpers, fetchRoundDiffEvidenceViaGh, the reply-text change, and an end-to-end addressFindings integration test)

- pnpm typecheck → clean

- pnpm test → 2942 vitest (+26 from the pre-existing 2916 baseline) + 440 Python unittest, all green, no regressions

## Verification Artifact

Not applicable — this is a backend/harness-logic change with no UI surface; the addresser.test.ts suite (including the two named regression fixtures) is the verification artifact.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a href="https://cursor.com/agents/bc-ae7b134d-418c-4c6c-acec-0711ac6c1cf9?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-ae7b134d-418c-4c6c-acec-0711ac6c1cf9&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

## Summary

Migrates the Ramp spend workflow from [AI-Builder-Team/ramp_lambda](https://github.com/AI-Builder-Team/ramp_lambda/tree/main/RampPipeline) into Surtr as two ECS pipelines:

- ramp-spend-pipeline: tracked-card discovery, raw and transformed weekly snapshots, merchant classification, financial metrics/chart, and researched cost-saving opportunities.

- ramp-superbuilders-report: strict input validation, active Superbuilders email rendering, duplicate-safe delivery claims, and SES delivery.

Only the active RampPipeline/superbuilders_app email is migrated. The retired finance_app dispatcher and its finance-only material-changes/risk-controls sections remain out of scope. The required Superbuilders cost-savings section is included.

## Card-to-project attribution

- Reconciles the live Ramp roster with Superbuilders.xlsx using stable cardholder_id values.

- Maps active-Q3 people with one canonical project in project_owners.py.

- Assigns missing, multi-project, inactive-Q3, or otherwise unclear cards to Unattributed.

- Preserves explicit historical labels for legacy card IDs.

Persisted reconciliation: 107 tracked cards = 81 active (55 mapped, 26 Unattributed) + 26 retired (24 explicit legacy, 2 Unattributed).

## Pipeline modes

ramp-spend-pipeline supports:

- Default: fetch and transform the full planned weekly window.

- mode=cards: roster snapshot.

- mode=transform: rebuild transformed snapshots from raw data.

- mode=classify: model-backed merchant classification.

- mode=metrics: financial metrics and spending chart.

- mode=cost: identify, web-research, validate, and prioritize cost-saving opportunities.

## Durable progress and retry safety

- Raw fetch and transform persist per week.

- Classification persists the merged cache after every completed merchant, so a later failed call does not discard prior work.

- Cost analysis checkpoints after identification, after every researched opportunity, and after prioritization.

- Cost checkpoints are bound to a fingerprint of the current merchant inputs; compatible retries resume only missing calls.

- The report validates matching week/year contracts and freshness before sending.

- Conditional, recipient-group-scoped S3 delivery claims prevent duplicate sends while allowing each approved group its own weekly delivery; successful and uncertain outcomes are persisted explicitly.

## Superbuilders email

The report includes only the active Superbuilders sections:

- Financial metrics and 12-week spending chart.

- Required researched cost-saving opportunity section.

- Project/category distribution.

The only approved groups are hard-enforced in code and SES destinations:

- schedule_1 (default): sanket.ghia@trilogy.com, ashwanth.r@trilogy.com.

- schedule_2: schedule 1 plus benji.bizzell@trilogy.com, david.harpur@trilogy.com, and ludel.mier@trilogy.com.

On-demand runs accept recipient_group=schedule_1|schedule_2 and default to schedule_1 when omitted. Both legacy cron slots remain disabled; the manifest is intentionally on-demand-only.

## Validation

- Ramp spend pipeline: 170 tests passed.

- Superbuilders report pipeline: 39 tests passed.

- Pipeline CDK manifest validation: 268 tests passed.

- Ruff passes on all modified Python files.

- Both ECS images build successfully; the report container imports its runtime modules.

- git diff --check passes.

## Production-data validation

- Refreshed and transformed all 30 planned weeks through week 31 of 2026 (10,341 transactions).

- Persisted 694 merchant classifications; the strict cost-readiness gate passes for 285 T4W merchants.

- Generated financial_metrics_w31_2026.json and the current spending chart.

- Generated the required week-31 cost artifact with $7,040 estimated monthly savings.

- Guarded email dry run passed all contracts and freshness checks.

- SES accepted one week-31 email to ashwanth.r@trilogy.com; the matching S3 delivery receipt is persisted as sent.

## Remaining operational step

Merge and deploy through the normal Surtr release process. Keep both legacy schedules disabled until explicitly approved; use on-demand recipient-group selection in the meantime.

## What this is

A read-only pull API for app customers checking data health, plus a third Pipeline API Keys tab on the APIs page for minting the keys that open it. Two GET endpoints. No push, no webhooks, no subscriptions, no new health computation, and no change to the alerting Surtr already has.

## The core property: this layer decides nothing

It is a serialization layer over records Surtr already keeps. There are no thresholds, freshness SLAs, row-count rules, null-rate checks or heuristics anywhere in it.

| Signal | Read from | Vocabulary returned |

| --- | --- | --- |

| Run outcome | run-record store — Redshift staging_other.pipeline_runs_prod + pipeline_registry_prod (Postgres fallback locally), via pipelineQueries | success · failed · partial · running · timeout — the runners' own values, lower-cased exactly as the query layer already does |

| Observer | DynamoDB surtr_pipeline_observations, via the observer store | OK · WARN · CRITICAL · UNAVAILABLE — the observer's own verdict, plus its own severities C/H/M/L |

Both are returned, side by side, never merged. An earlier revision of this branch had a single state field that blended them using the dashboard's bucketing rules — that was this layer inventing a judgement Surtr never made. The two signals are independent and can legitimately disagree (a run succeeds while the observer flags the data; a run fails while the last observation still says OK), so last_run.status and observer.verdict are reported separately and unmapped. There is no mapping table anywhere in the diff.

No signal is never "healthy." Never observed, observation switched off for the pipeline, and observation store unreadable all report Surtr's own UNAVAILABLE with a distinguishable unavailable_reason. A pipeline that has never run reports last_run: null. Nothing silently defaults to green.

## Endpoints

| Requested | Implemented |

| --- | --- |

| GET /pipelines | GET /v1/pipelines |

| GET /pipeline/<id> | GET /v1/pipeline/{id} |

/v1 is the repo's existing public-read prefix and already has a Next.js rewrite to the Hono backend. A bare /pipelines rewrite would have shadowed the existing /pipelines UI page, so the prefix isn't optional. These two routes are explicitly exempted from the /v1 SURTR_API_KEYS shared-secret gate and authenticate with their own pak_ keys — tested in both directions.

### GET /v1/pipelines

?limit= (default 50, max 200) &offset= &status= (run outcome) &verdict= (observer) &name= (id or name prefix)

{
"as_of": "2026-08-04T01:34:43.159Z",
"count": 2, "total": 2, "limit": 2, "offset": 0,
"run_status_counts": { "success": 1, "failed": 1 },
"data": [
{
"id": "aws-bedrock-token-metrics",
"name": "AWS Bedrock Token Metrics Pipeline",
"description": "Fetches Bedrock token metrics by model and loads them into the warehouse",
"schedule": {
"expression": "cron(0 7 * * ? *)", "enabled": true, "timezone": "UTC",
"next_run_at": "2026-08-04T07:00:00.000Z"
},
"last_run": {
"run_id": "run-9f2c1a", "status": "success",
"started_at": "2026-08-04T01:28:43.159Z",
"completed_at": "2026-08-04T01:32:17.159Z",
"duration_ms": 214000, "trigger_type": "schedule", "error": null
},
"observer": {
"verdict": "WARN", "unavailable_reason": null,
"last_evaluated_at": "2026-08-04T01:34:43.159Z",
"open_finding_count": 2, "worst_severity": "H", "window_days": 7
}
},
{
"id": "gcp-billing-pipeline",
"name": "GCP Billing Pipeline",
"description": "Loads the GCP billing export into the warehouse",
"schedule": { "expression": "rate(6 hours)", "enabled": true, "timezone": "UTC",
"next_run_at": "2026-08-04T07:28:43.159Z" },
"last_run": {
"run_id": "run-3b7e", "status": "failed", "duration_ms": 12000,
"trigger_type": "schedule",
"error": "BigQuery export table not found: billing_export_v1"
},
"observer": {
"verdict": "UNAVAILABLE",
"unavailable_reason": "No observation recorded in the last 7 days.",
"last_evaluated_at": null,
"open_finding_count": 0, "worst_severity": null, "window_days": 7
}
}
]
}

Note the second row: the run failed and the observer has nothing to say. Both facts are visible; neither is resolved into a verdict.

### GET /v1/pipeline/{id}

?runs= (default 20, max 100)

{
"as_of": "2026-08-04T01:34:43.159Z",
"id": "aws-bedrock-token-metrics",
"name": "AWS Bedrock Token Metrics Pipeline",
"description": "Fetches Bedrock token metrics by model and loads them into the warehouse",
"schedule": { "expression": "cron(0 7 * * ? *)", "enabled": true, "timezone": "UTC",
"next_run_at": "2026-08-04T07:00:00.000Z" },
"last_run": { "run_id": "run-9f2c1a", "status": "success", "duration_ms": 214000, "error": null },
"runs": [
{ "run_id": "run-9f2c1a", "status": "success",
"started_at": "2026-08-04T01:28:43.159Z", "completed_at": "2026-08-04T01:32:17.159Z",
"duration_ms": 214000, "trigger_type": "schedule", "error": null },
{ "run_id": "run-8a11", "status": "failed",
"started_at": "2026-08-03T01:28:43.183Z", "completed_at": "2026-08-03T01:29:24.183Z",
"duration_ms": 41000, "trigger_type": "schedule",
"error": "Throttled by CloudWatch GetMetricData (429) after 5 retries" }
],
"observer": {
"verdict": "WARN", "unavailable_reason": null, "enabled": true,
"score": 86,
"summary": "Loaded, but the newest partition lags a day and volume dipped.",
"last_evaluated_at": "2026-08-04T01:34:43.159Z",
"last_evaluated_run_id": "run-9f2c1a",
"window_days": 7,
"findings": [
{
"category": "freshness",
"title": "Latest partition is 26h old",
"severity": "H",
"evidence": "max(usage_date) = 2026-08-03 vs run date 2026-08-04",
"recommendation": "Confirm the upstream export completed before the 07:00 window.",
"occurrences": 2,
"first_fired_at": "2026-08-03T01:28:43.159Z",
"last_fired_at": "2026-08-04T01:28:43.159Z",
"last_fired_run_id": "run-9f2c1a"
},
{
"category": "volume",
"title": "Row count 18% below 7-day median",
"severity": "M",
"evidence": "18,442 rows vs median 22,600",
"recommendation": "Check whether a linked account dropped out of the scan.",
"occurrences": 1,
"first_fired_at": "2026-08-04T01:28:43.159Z",
"last_fired_at": "2026-08-04T01:28:43.159Z",
"last_fired_run_id": "run-9f2c1a"
}
],
"ignored_findings": [
{ "category": "cosmetic", "title": "Deprecation warning in boto3",
"reason": "known, harmless", "ignored_at": "2026-07-26T20:28:43.183Z" }
],
"history": [
{ "run_id": "run-9f2c1a", "verdict": "WARN", "score": 86, "evaluated_at": "2026-08-04T01:34:43.159Z" },
{ "run_id": "run-8a11", "verdict": "OK", "score": 100, "evaluated_at": "2026-08-03T01:28:43.159Z" }
]
}
}

Both samples are real output from the routes, not hand-written.

## ⚠️ The observer records no per-check thresholds or observed values

The spec asked for, per observer check: *what it checks · current verdict · configured threshold · observed value · when it last evaluated · when it last fired.* Four of those exist; two do not, and I did not invent them.

Surtr's observer is not a set of standing checks with configured thresholds. It is a per-run LLM evaluation that emits findings, each with severity · category · title · evidence (free text) · recommendation. There is no per-check threshold and no structured observed value anywhere in the store.

| Asked for | Returned as | Source |

| --- | --- | --- |

| what it checks | category + title | the finding |

| current verdict | severity per finding, verdict overall | the finding / observation |

| configured threshold | omitted | does not exist |

| observed value | closest is evidence (the observer's own free-text measurement, e.g. "18,442 rows vs median 22,600") | the finding |

| when it last evaluated | observer.last_evaluated_at | observation |

| when it last fired | last_fired_at (+ first_fired_at, occurrences) | the observations the finding appeared on |

I also dropped the thresholds block the earlier revision exposed — it was the observer's *global* scoring rubric (severity→deduction weights and verdict bands), not a per-check threshold, and shipping it under that name would have implied a precision the observer doesn't have.

If you want real per-check thresholds and observed values, that's a change to what the observer records, not to this API — say the word and I'll scope it separately.

## Because the audience is external

Everything below is deliberately omitted from both endpoints. There's a test that greps the raw detail response for each of these strings and fails if any appears:

| Omitted | Why |

| --- | --- |

| step_function_arn, step_function_url | internal topology |

| cloudwatch_log_group, cloudwatch_logs_url, log_stream | internal topology |

| AWS account ids, console links | internal topology |

| owners (name + email) | internal staff identities |

| triggered_by | can be an internal user id |

| output_summary | names internal warehouse tables (core_finance.…) |

| model_id, observer_version, braintrust span ids | internal observer implementation |

| deployed_at | internal deploy metadata; doesn't answer "can I trust this data right now?" |

| run input_params | can carry caller-supplied values |

Kept, as a judgement call: run_id (a Surtr domain id, not infra — it's the join key between the run history and last_fired_run_id, and useful in a support conversation) and trigger_type (schedule / manual — explains *why* a run happened, reveals nothing). Tell me if you'd rather either went.

Error text is the pipeline's own failure reason — unwrapped from its orchestration envelope, cut before any stack trace or SQL dump, capped at 500 chars, and with any surviving infrastructure identifier redacted. See the section below: this was a real leak, not a precaution.

## 🔴 A real leak, found by running it against live data

Calling the endpoints locally against the live registry surfaced something the unit tests could not: Step Functions task failures are stored as States.TaskFailed: {<the entire ECS task description>}. The original stack-frame trim didn't touch a single-line JSON blob, so hubspot-admissions-funnel and quickbooks-ap-sync were serving subnet ids, ENI ids, MAC addresses, private IPv4s, ip-…​.ec2.internal hostnames, cluster and task ARNs, the ECR image URI and the AWS account id to whoever held a key.

Fixed in 36a11e48 and a600d2c8. Failures are now unwrapped rather than trimmed:

- a Lambda error (RuntimeError: {"errorMessage": …, "stackTrace": […]}) yields its errorMessage — the pipeline's own reason, which is what a customer wants;

- a Step Functions blob contains no reason at all, only topology, so it is dropped and only States.TaskFailed survives;

- a blob with neither says the detail was omitted, which beats a null that would read as "no error" on a failed run;

- then first line only, then ARN / subnet- / eni- / i- / ECR URI / private IP / MAC / account-id redaction, then the length cap.

The second commit exists because the runners interpolate Redshift errors containing bare double quotes into the payload without re-escaping, so JSON.parse refuses the blob and five real pipelines lost their reason to a bare "RuntimeError". errorMessage is now salvaged from the malformed text.

Verified against all 90 live pipelines: every one of arn:aws, subnet-, eni-, 12-digit account ids, private IPv4 ranges, MACs, ec2.internal, /aws/lambda/, /aws/ecs/ and dkr.ecr scans clean, while the ten currently-failing pipelines still report their own reasons (docker source is stale: max=2026-W29, lag=3 weeks, relation "…" does not exist, TimeoutError, …).

## Contract details

- snake_case throughout, matching the specified as_of, and matching the repo's other public JSON.

- as_of on every response — the newest instant Surtr computed anything reflected in the payload (max(last run's completed_at, observer's last_evaluated_at)), never the serve time. Per-item freshness is also visible via last_run.completed_at and observer.last_evaluated_at.

- ISO-8601 UTC everywhere; durations are duration_ms consistently.

- Enum values are Surtr's, and ?status= / ?verdict= validate against those same lists, so a filter value and a serialized value can never drift.

- run_status_counts is the cheap convenience rollup (from the registry read we already do). Run outcomes only — an observer rollup would force a fan-out over every pipeline, and this is explicitly a convenience.

## Key minting — third tab

APIs → Pipeline API Keys (pak_…), built from the same components and the same mint / copy-once / revoke flow as the existing Gateway tab, with scope cards where that tab has source cards: a pipelines:read scope card, label + optional expiry, one-time gold panel with Copy / Dismiss and a ready-to-paste curl, and a key table with scope chips and Revoke. A load failure renders "Failed to load keys …" and explicitly not "no keys".

The picker offers exactly one scope, because exactly one capability exists. An earlier revision also listed a greyed-out "Trigger / modify pipelines · soon" card; it was removed — a placeholder advertising an API nobody is building is worse than no entry.

Storage mirrors gateway_keys exactly: pipeline_api_keys in Postgres, only the sha256 of the raw key persisted, secret returned once. tRPC gains listPipelineApiScopes / listPipelineApiKeys / createPipelineApiKey / revokePipelineApiKey, all api_admin-gated.

I did not invent a scope system — I mirrored the gateway's. A separate table (rather than reusing gateway_keys) is deliberate: a gwk_ key can never open the pipeline API, a pak_ key can never open the gateway, and each surface stays independently revocable. No key from another tab grants access, and neither does a SURTR_API_KEYS shared secret.

## Unhappy paths

Gateway-style envelope, { "error": "<code>", "message": "…" }:

| Case | Status |

| --- | --- |

| no key | 401 unauthorized |

| unknown / revoked / expired key (indistinguishable, on purpose) | 401 unauthorized |

| key-store read throws | 401 unauthorized (fails closed) |

| live key without pipelines:read | 403 forbidden |

| unknown pipeline id | 404 not_found |

| bad status / verdict / limit / offset / runs | 400 bad_request |

| pipeline registry not configured | 503 service_unavailable |

No silent empty-200s. The 401 body names the header and never echoes the presented key.

## Tests

56 new tests, none touching AWS/Redshift/Postgres — the key validator and the observation reader are injected via the same seam the gateway tests use.

test/pipeline-api/pipeline-api.test.ts (46) · test/pipeline-api/schedule.test.ts (10) · test/ui/pipeline-apis-tab.test.tsx (5)

Beyond the happy paths, 404, 401, 403 and 503, the ones worth calling out:

- a failed run alongside an OK observer verdict returns both, unmerged

- the response has no state / health / top-level status field at all

- never-observed → UNAVAILABLE + reason, not OK

- observation store throws → UNAVAILABLE + the store's reason, and the run outcome is unaffected

- ignore-list read throws → UNAVAILABLE, never "no findings"

- operator-disabled observer → UNAVAILABLE + enabled: false

- the raw detail response contains none of ~10 internal strings (ARNs, log groups, account id, owner email, Clerk user id, model id, warehouse table name)

- the observer block has no thresholds / scoring, and no finding has threshold / observed_value

- as_of is older than the serve time and equals the newest underlying computation

- stack-trace stripping (Python and Java frames), blank-line truncation, 500-char cap

- both cross-surface auth-leak directions

### Results — actually run

| Gate | Result |

| --- | --- |

| pnpm lint (biome) | pass — 95 files, no fixes applied |

| pnpm build (tsc) | pass — clean |

| pnpm test:unit (what CI runs) | pass — 45 files, 653 tests |

| pnpm build:ui (next build) | pass |

| pnpm test (full suite) | 1263 pass / 7 fail |

The 7 failures are pre-existing on origin/main — all in test/connectors/redshift.test.ts (mapSiteRow), which CI's test:unit excludes. Confirmed by stashing this branch and re-running: identical 7 failures, 1212 passing.

## Deploy notes

- Migration required. New pipeline_api_keys table — pnpm db:push, or the equivalent applied through ECS Exec in prod (this repo has no auto-migrate). Until it exists the endpoints 401 every key; nothing else is affected.

- No new secrets, env vars or IAM. No key material is logged.

- Routes register only when the Postgres db is present — same posture as the gateway, absent rather than open.

## Deliberately left out

- No write surface, and no placeholder for one. pipelines:read is the only scope that exists. A key whose scopes don't cover a route still gets a 403 — that branch is tested with a scope-less key.

- No third resource, no push/webhooks/subscriptions, and no change to existing alerting.

- Cursor pagination — limit/offset like the gateway; the registry read is a full-set read paginated in memory.

- Cron edge syntax — next_run_at handles every expression the deployed pipelines actually use (wildcards, lists, ranges, steps, day/month names). EventBridge L / W / # return null rather than a guess; no deployed pipeline uses them.

- Cost note. ?verdict= is the only parameter that needs observer records for pipelines outside the page (one DynamoDB query each, the same fan-out the dashboard already does every load). Every other request reads the observer for the page only.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Summary

- Refresh the branch from current main and renumber the stale spec from 11 to 12.

- Convert the legacy mock-user dictionary into UserPermissions and override _require_auth, the dependency AWS Spend endpoints actually use.

- Make _create_budget_sim_client delegate to the canonical test-client factory.

- Refresh stale test setup exposed after auth succeeds: use UserPermissions in direct BU-access tests, isolate summary tests from persisted Redis cache state, and provide the required WoW heatmap grand-total mock.

- Remove the obsolete tests/openai rename from scope because tests/openai_spend already exists on main.

## Root cause

create_test_client_with_user overrode get_user_from_clerk, but the router authenticates through _require_auth → get_current_user_permissions. The override was never used, so real auth returned 401 No valid Authorization header across 45 test call sites.

## Impact

This is a test-only developer-experience fix. Production authentication and endpoint behavior are unchanged.

## Validation

- uv run ruff format tests/test_aws_spend_router.py

- uv run ruff check tests/test_aws_spend_router.py

- uv run pytest tests/test_aws_spend_router.py -q --timeout=60 — 129 passed

- Repeated the complete pytest command to verify cache independence — 129 passed again

## Linear

KLAIR-2872

## Demo

<img width="2078" height="1481" alt="image" src="https://github.com/user-attachments/assets/f2a39079-847b-4f93-a17b-19aeccbbc76e" />

<img width="2027" height="1561" alt="image" src="https://github.com/user-attachments/assets/ac472448-1958-4075-abd1-2da291695c03" />

## Summary

- replace individual Education on-demand choices with the server-owned Physical Private Schools and Other Education Verticals report groups

- generate one consolidated group document with group commentary/action items plus compact per-member P&L breakdowns and explicit missing-data states

- preserve group membership and unavailable-member audit data through orchestration/job summaries while keeping group names in Drive, ledger, results, and delivery emails

- disable Education generation in weekly and monthly scheduled QTD entry points without changing Software/CF scheduling

## Why

Education QTD generation previously treated every Education BU as a separate report, ledger result, and delivery. Finance needs at most two meaningful consolidated reports while retaining member-BU visibility and server-controlled dynamic membership.

Closes #3455

## Validation

- ANTHROPIC_API_KEY=test uv run pytest tests/monthly_qtd_report/ tests/routers/test_qtd_ondemand_router.py tests/crons/test_ondemand_qtd_report_cron.py --ignore=tests/monthly_qtd_report/test_qtd_reports_router.py -q — 795 passed

- targeted frontend Vitest suite — 37 passed

- pnpm lint:pr

- pnpm tsc --noEmit

- Ruff formatting/checks and Pyright on changed backend files

test_qtd_reports_router.py is excluded from the broad local run because importing the full app requires Zendesk credentials; the affected on-demand router suite passes independently.

## Generated report proof

- [Physical Private Schools — Q3 FY2026 QTD BvA](https://docs.google.com/document/d/1qdXACOSYVih3tXyii5MSRyBpJlDKcaGNpOkAXWXV-FA/edit)

- [Other Education Verticals — Q3 FY2026 QTD BvA](https://docs.google.com/document/d/1LOvDTGv8v0hFVClQsy7jJI5at9ht2T6yqa6O5JLDJ1Q/edit)