Automated fix for openai-usage-pipeline — fix_class code_fix, scope tier draft.
Resolves https://github.com/AI-Builder-Team/Surtr/issues/1300
> Ready for review — verification is green; HEIMDALL_READY_PRS opens verified tier-draft fixes ready for review. A human still merges — auto-merge never applies outside tier auto.
## What's broken
Run b34335fc-2125-43b4-ac80-b191f695cebe of openai-usage-pipeline processed 28 BUs and published 573 usage rows, but the /costs line-item fetch for Trilogy-Academics and Trilogy-Skyvera exhausted all 5 retries against OpenAI's org rate limit — offending line: [ERROR] Line-item cost fetch failed for BU Trilogy-Skyvera, API key 1: 429 Client Error: Too Many Requests for url: https://api.openai.com/v1/organization/costs?..., with body You've exceeded the 30 request(s) every 1 minute(s) rate limit. Both BUs' usage rows were persisted with billed_cost_dollars=$0 (Trilogy-Academics 106 rows, Trilogy-Skyvera 0 rows) and the run was recorded outcome=partial; 30+ 429 WARNING lines across the run show sustained rate-limit pressure, not a one-off spike. This is not row loss — the usage rows are correct and the code (handler.py, added in PR #1161) deliberately degrades to $0 billed cost expecting the T-2 re-pull to self-heal — but the billed-dollar (spend/Services) figures for those two BUs are wrong until a later run succeeds.
Root cause. The client has no cross-request rate limiter: _request_with_retries in src/openai_client.py issues every call as fast as the network allows, and the only throttle — a 0.5s post-response sleep gated on REQUEST_DELAY_SECONDS inside each individual fetch loop — does not smooth bursts across the ~150 sequential requests the run makes (per BU: one /usage call, one /api_keys call per project, one /projects name call per project, and one /costs call). Processing 28 BUs back-to-back therefore overruns OpenAI's evidenced org quota of 30 requests per 60 seconds, and by the time later BUs (Trilogy-Academics, Trilogy-Skyvera) reach their /costs call the rolling window is saturated, so the 2/4/8/16/32s backoff of the 5-retry budget (~31s of waiting) cannot clear the per-minute window and the fetch is abandoned. The failure lands on /costs specifically because it is the last per-BU call, after the usage/api_keys/names calls have already consumed the window.
## What this PR changes
Add a process-wide token-bucket / minimum-interval rate limiter to src/openai_client.py, applied inside _request_with_retries so it governs every OpenAI call (usage, api_keys, project names, and costs) uniformly, sized to the quota the log states — 30 requests per rolling 60 seconds — and made env-configurable (e.g. OPENAI_MAX_REQUESTS_PER_MINUTE, default 30) with the value declared in pipeline.json's environment block if a non-default is wanted. Add a unit test in tests/test_openai_client.py that asserts the limiter spaces a burst of calls to stay under the cap (monkeypatching time.sleep/monotonic so it runs fast). Keep the change confined to the pipeline's src/ and tests/: do NOT add automatic re-pull logic or alter the cron schedule (PR #1188 already staggered the 06:00/07:00 quota window) and do NOT rewrite the existing graceful-degradation path in handler.py — the token bucket is the smallest change that removes the root cause.
Why this fixes it. A shared token bucket is exactly the remediation the observer recommends ('a shared token-bucket across all BUs') and it is fully implementable within Tier A (pipelines/runners/openai-usage-pipeline/), so no scope widening or human-only judgement is required; the quota to enforce is not guessed but read directly from the 429 body ('30 request(s) every 1 minute(s)'), which removes the main tuning risk. It is timeout-safe: the run used 394s of the 900s budget and roughly 120s of that was wasted 429 backoff (five BUs × up to 31s of retry sleeps), so smoothing requests to the 30/min ceiling is approximately runtime-neutral — it trades burst-then-stall for steady pacing rather than adding net wall-clock. This prevents the recurring silent-dollar corruption at the source instead of relying solely on the T-2 self-heal, while the already-merged degradation path (PR #1161) remains the safety net for any residual failure.
### Files changed
.../runners/openai-usage-pipeline/pipeline.json | 3 +-.../openai-usage-pipeline/src/openai_client.py | 65 ++++++++++++++++++++++
.../tests/test_openai_client.py | 53 ++++++++++++++++++
3 files changed, 120 insertions(+), 1 deletion(-)
## Verification
### pytest (pipelines/runners/openai-usage-pipeline/tests) — exit 0
``
ic_missing_insert_rowcount_raises PASSED [ 81%]
tests/test_redshift_handler.py::TestAtomicPublish::test_atomic_empty_rows_with_owned_windows_still_deletes PASSED [ 82%]
tests/test_redshift_handler.py::TestAtomicPublish::test_atomic_owned_windows_merge_with_row_derived_pairs PASSED [ 82%]
tests/test_redshift_handler.py::TestAtomicPublish::test_atomic_invalid_owned_windows_are_skipped PASSED [ 83%]
tests/test_redshift_handler.py::TestAtomicPublish::test_empty_rows_without_owned_windows_is_a_noop PASSED [ 84%]
tests/test_secrets.py::TestGetOpenAiBuKeys::test_returns_bu_key_mapping PASSED [ 85%]
tests/test_secrets.py::TestGetOpenAiBuKeys::test_normalizes_single_key_to_list PASSED [ 85%]
tests/test_secrets.py::TestGetOpenAiBuKeys::test_raises_on_secrets_manager_error PASSED [ 86%]
tests/test_write_modes.py::TestWriteModes::test_old_mode_has_zero_secondary_side_effects PASSED [ 87%]
tests/test_write_modes.py::TestWriteModes::test_dual_mode_primary_first_then_secondary_lane_then_ledger PASSED [ 87%]
tests/test_write_modes.py::TestWriteModes::test_dual_mode_secondary_failure_is_partial_and_primary_intact PASSED [ 88%]
tests/test_write_modes.py::TestWriteModes::test_dual_mode_ledger_failure_is_partial PASSED [ 89%]
tests/test_write_modes.py::TestWriteModes::test_new_mode_writes_only_secondary_and_failures_raise PASSED [ 90%]
tests/test_write_modes.py::TestWriteModes::test_run_id_falls_back_to_lambda_request_id PASSED [ 90%]
tests/test_write_modes.py::TestWriteModes::test_dual_mode_incomplete_run_is_never_ledgered_as_published PASSED [ 91%]
tests/test_write_modes.py::TestWriteModes::test_invalid_mode_fails_loud PASSED [ 92%]
tests/test_write_modes.py::TestValidEmptyConvergence::test_valid_empty_fetch_converges_window_and_ledgers_zero_published[dual] PASSED [ 92%]
tests/test_write_modes.py::TestValidEmptyConvergence::test_valid_empty_fetch_converges_window_and_ledgers_zero_published[new] PASSED [ 93%]
tests/test_write_modes.py::TestValidEmptyConvergence::test_failed_bu_window_is_never_deleted[dual] PASSED [ 94%]
tests/test_write_modes.py::TestValidEmptyConvergence::test_failed_bu_window_is_never_deleted[new] PASSED [ 95%]
tests/test_write_modes.py::TestValidEmptyConvergence::test_no_bus_path_has_no_secondary_side_effects PASSED [ 95%]
tests/test_write_modes.py::TestLedgerModule::test_record_publication_inserts_row PASSED [ 96%]
tests/test_write_modes.py::TestLedgerModule::test_record_publication_ …_(truncated)_
<details>
<summary>Run metadata</summary>
| Field | Value |
| --- | --- |
| Pipeline | openai-usage-pipeline |
| Failing run | b34335fc-2125-43b4-ac80-b191f695cebe |
| Occurrence | 1 (times this exact failure signature has been seen) |
| Signature | 891d836c3c73ade63fa643dbdab2eb22dd736f97a36e6807df9b27806f4df31a |
| Verify | green |
</details>
---
🤖 Opened by heimdall. mercy reviews this PR automatically; heimdall revises on REQUEST_CHANGES (bounded rounds). Tier-auto PRs may auto-merge on mercy approval when the consumer enables it; everything else waits for a human. Mention heimdall in a comment to direct it, or add the manual-dev` label to take the PR over and stop it entirely.