## Summary
Adds Surtr's control-and-wait adapter for the existing Raptor EC2 collector, plus disabled daily scheduling configuration. Tracks [AI-1033 — trigger and monitor the collector](https://linear.app/builder-team/issue/AI-1033) and [AI-1037 — scheduling and operations](https://linear.app/builder-team/issue/AI-1037), under [AI-1029 — Raptor refresh through Surtr](https://linear.app/builder-team/issue/AI-1029).
Production effect: dormant/additive preparation. Both dashboard/API manual execution (on_demand_enabled: false) and the scheduled rule are disabled. The existing framework rejects ordinary manual executions before run-record creation or ECS work. This PR does not activate collection; the first production invocation requires the remaining host/controller wiring, an explicit reviewed enablement and live acceptance below. Existing IAM remains the trust boundary for principals able to submit system-trigger events directly.
### Raptor dependency status — verified 2026-11-02
| Dependency | Current state |
| --- | --- |
| Raptor history and collector code | [Raptor PR #5](https://github.com/AI-Builder-Team/Raptor-DSS/pull/5) and [PR #6](https://github.com/AI-Builder-Team/Raptor-DSS/pull/6) merged into 0e942fe237131448e83b0105a560b25f26fb9719. Its tree matches the independently reviewed candidate. |
| Production DSS reader | [Deployment 41505491935](https://github.com/AI-Builder-Team/Raptor-DSS/actions/runs/41505491935) succeeded. Task raptor-dss:4 healthy; exact reviewed image sha256:66eb47e5e4b791831c563cc305c8fdab528db2ed5e007f7d6f16168abe0fb686. Effective reader policy and boundary v3 verified. Independent POST review passed. |
| Existing data served by DSS | All six authenticated model reads passed. Accepted publication and source clocks unchanged. No new collection was performed; absent refresh-attempt state remains null. Temporary validation identity revoked and subsequent 401 confirmed. |
| EC2 collector package | Installed and verified inactive under /opt/raptor-dss-dev-20261030/collector/refresh-dev-20261102/raptor-release-0e942fe. Actual-host file/dependency hashes, pinned Node 22 and offline entrypoint import passed. Host is stopped. |
| Surtr integration | Surtr activation is held. Real single-flight table/configuration, active host entrypoint, SSM document, remaining collector permissions and live trigger-to-DSS acceptance are still required. The reviewed code adds an explicit manual-execution hold rather than relying on the disabled schedule alone. |
The DSS deployment is complete; it must not remain listed as a pending dependency. Collector staging is not proof of an active Surtr refresh path.
## Why
Raptor's working browser collectors must stay on their existing EC2 runtime. Surtr should start and monitor that work using its normal execution, ownership and notification mechanisms, without moving source records into Redshift or rebuilding collection. Persisted execution identity and terminal verification prevent an uncertain dispatch or successful SSM response from being mistaken for a completed refresh.
## Business Value
- Refresh selected datasets without recollecting unrelated successful work.
- Reuse saved collections for recovery, with an explicit fresh-collection mode.
- Use existing Surtr failure/partial notifications and per-dataset cadence configuration.
- Keep sensitive records in private S3 and authenticated DSS; retain standard operational run metadata in Surtr.
## How does it work
1. src/main.py validates dataset selection, recovery/fresh mode and deployment-owned cadence configuration before AWS work. Supported datasets are building, visit_event, alert_counts, user_contact and daily_activity; personnel is excluded.
2. src/controller.py uses Surtr's single-flight table to bind the request to one execution and owner. It records dispatch intent before calling the fixed SSM document on the existing EC2 instance; uncertain acknowledgement is reconciled instead of blindly redispatched.
3. The existing host collector performs collection, validation and private publication. Surtr receives operational results, not source records, credentials or raw remote errors.
4. The adapter checks execution identity, requested datasets, actual terminal results and owned-process cleanup. It maps complete, partial/continuing and failed outcomes to standard Surtr results. The ordinary result finalizer and notifications remain unchanged.
5. pipeline.json prepares a daily kickoff at 04:00 fixed EST / 09:00 UTC, with separately configurable dataset cadences. Both on_demand_enabled and schedule.enabled are false; no custom scheduler, alerts or UI are introduced. Enable each through reviewed configuration only after its activation gates pass.
## Scope
### Included in this phase
- Raptor-specific ECS control adapter, owner registration, fixed SSM/host permission declarations, disabled scheduling and operator documentation.
- Exact final diff paths:
pipelines/owners.jsonpipelines/runners/raptor-refresh/Dockerfile
pipelines/runners/raptor-refresh/OPERATIONS.md
pipelines/runners/raptor-refresh/README.md
pipelines/runners/raptor-refresh/deployment/host-permissions.json
pipelines/runners/raptor-refresh/deployment/ssm-document.json
pipelines/runners/raptor-refresh/pipeline.json
pipelines/runners/raptor-refresh/src/controller.py
pipelines/runners/raptor-refresh/src/main.py
pipelines/runners/raptor-refresh/src/requirements.txt
pipelines/runners/raptor-refresh/tests/test_controller.py
### Deliberately excluded for later phases
- Production wiring and live acceptance: bind the real Surtr lease table/configuration and reviewed collector permissions; install the version-pinned SSM document against the accepted active host entrypoint. The document currently declares /opt/raptor-dss-dev-20261030/collector/refresh/host-bridge.py, while the verified package is staged in the separate release directory above. Staging did not populate that active path or install its config. Resolve and verify this binding before any dispatch; do not assume the staged package is already callable.
- Schedule activation: only after a real targeted trigger reaches private publication and authenticated DSS readback, and recovery/history continuation plus owned-resource cleanup pass. Keep manual validation notifications suppressed using the existing run option.
- Shared Surtr scheduler, alerting, finalizer or UI changes; warehouse ingestion of Raptor records; personnel refresh; collector/browser rewrites; manual date-range backfills; recipient-key issuance.
### Approved conventions and review exceptions
- Private S3/DSS replaces warehouse staging/Core/marts for sensitive source records. Ordinary Surtr inputs, operational summaries and errors remain allowed through the standard metadata path; no pipeline-specific metadata allowlist framework.
- Keep the accepted report-based collectors rather than replacing them with underlying-source acquisition. Original-response retention and recovery/cleanup remain Raptor-side responsibilities, not waived requirements.
- Compatible successful updates may publish alongside valid older components, preserving each component's actual source time and provenance.
- The user explicitly approved the test-LOC exception for this Surtr PR and the Raptor collector PR on 2026-11-02. Including the focused activation-hold regression, this PR has 275 / 1,138 added code/config LOC = 24.165% tests, including helpers and excluding Markdown. Total diff is 1,441 net added lines, below the approximately 3,000-line Surtr goal. No CI, security, independent-review or release waiver.
- Permission JSON declares intended grants; it does not prove those grants are applied. Raptor's applied reader permissions do not supply the controller/collector's remaining permissions.
## Test plan
### Automated validation
- Focused controller suite: 8/8 passed, python3 -m unittest discover -s pipelines/runners/raptor-refresh/tests -v. Covers the actual configuration hold, input validation, result accounting, uncertain dispatch, cancellation fencing, stopped-host readback and predecessor ordering; no source acquisition. The new configuration test failed before the repair because the missing flag defaults true, then passed with the explicit false value.
- Existing framework guard regression: 1 focused test passed; 30 unrelated tests filtered, npm test -- --runInBand test/constructs/ecs-step-function.test.ts -t 'rejects disabled ECS execution before creating a run record and audits it' in pipelines/cdk. Synthesized guard routes ordinary manual input to OnDemandDisabled before run-record creation/ECS. Changed-test Ruff lint/format and git diff --check pass. No shared framework code/test was changed.
- [Exact-head CI 41501334204](https://github.com/AI-Builder-Team/Surtr/actions/runs/41501334204): all seven jobs passed at d13761bb710310f387d5c73b54e0fb19c3bc50f6, including Ruff, Biome, CDK, pipeline Lambdas, pipeline runners, Surtr tests and typecheck. Inspect with gh pr checks 2234 --repo AI-Builder-Team/Surtr.
- [Earlier Mercy workflow 41501334591](https://github.com/AI-Builder-Team/Surtr/actions/runs/41501334591) completed successfully; that workflow status is not current review approval. The [subsequent review](https://github.com/AI-Builder-Team/Surtr/pull/2234#pullrequestreview-5741965869) requested changes at the same head; its blocking finding is addressed in the final clarification section below. Independent code review passed at 3d216356a07ec396bf7e37272fb7f812cbdf537e; final Ruff-only commit d13761bb independently verified AST-equivalent. Earlier formatting failure was repaired, not waived.
- git diff --check 60008084...HEAD and exact diff scope inspection against that baseline; no production behavior changes in this PR-description update.
- Raptor's live six-model reader proof is linked above as completed dependency evidence, not as Surtr end-to-end acceptance. No actual Surtr dispatch, present refresh-status readback or scheduled run is claimed here.
### Time for Implementation
Estimated 3–5 engineering days without AI assistance for this adapter/configuration slice, integration investigation, review and deployment validation; not a measured duration or delivery promise, and excludes rebuilding the existing Raptor collectors.
## Review repairs and contract clarifications
### Blocking environment-variable finding — false premise; no code repair
Review: [5741965869](https://github.com/AI-Builder-Team/Surtr/pull/2234#pullrequestreview-5741965869), [blocking thread](https://github.com/AI-Builder-Team/Surtr/pull/2234#discussion_r4446641190). Existing ownership: [AI-1033](https://linear.app/builder-team/issue/AI-1033). Reviewed/current head: d13761bb710310f387d5c73b54e0fb19c3bc50f6; base: 60008084db3809d648fcfd8f1326a546327796d7. No repair ticket or commit is required for this disproved premise.
The finding assumes that variables omitted from pipeline.json.environment are absent from the deployed task. The supported caller is Surtr's standard ECS Step Functions execution, not a raw container launched without that framework. Required variables are assembled at two existing framework boundaries:
1. [Raptor configuration](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/runners/raptor-refresh/pipeline.json#L5-L13) selects ECS and sets single_flight: true. [EcsPipeline](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/cdk/lib/constructs/ecs-pipeline.ts#L164-L177) therefore creates the lease table, then [adds SINGLE_FLIGHT_TABLE to the container environment](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/cdk/lib/constructs/ecs-pipeline.ts#L188-L215).
2. [EcsStepFunction.createEcsTask](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/cdk/lib/constructs/ecs-step-function.ts#L235-L252) supplies RUN_ID from $.run.run_id, serialized PARAMS from $.params, and the standard RUN_RESULTS_BUCKET and RUN_RESULTS_PREFIX constants.
3. Those values are [passed as container overrides to EcsRunTask](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/cdk/lib/constructs/ecs-step-function.ts#L291-L305). The [ordinary non-fanout execution](https://github.com/AI-Builder-Team/Surtr/blob/d13761bb710310f387d5c73b54e0fb19c3bc50f6/pipelines/cdk/lib/constructs/ecs-step-function.ts#L1681) uses this same constructor. The overrides supply the run-specific variables; the task definition supplies the lease-table variable and Raptor's declared cadence map.
Thus the claimed KeyError from absent declarations does not follow on the supported deployment path. Duplicating these framework-owned values in pipeline configuration would not fix a missing boundary; it would duplicate values already supplied by the framework. This is direct current-head data-flow evidence, not an inference from green CI or disabled scheduling.
Validation remains 7/7 focused controller tests and seven passing required CI jobs at the unchanged head, as recorded above. No new test is manufactured for the disproved missing-injection premise. Mercy's subsequent review 5742252815 explicitly withdrew the environment-variable finding. This does not constitute overall approval: its new host-binding blocker is addressed below. Both nonblocking findings remain untouched.
### Host-path activation blocker — bounded repair under AI-1038
Latest review 5742338051, blocking comment 4446925599, reviewed head d13761bb710310f387d5c73b54e0fb19c3bc50f6. Assigned repair: [AI-1038](https://linear.app/builder-team/issue/AI-1038), Cainã Perotti Barbosa. The accepted defect was exposing manual invocation before the declared host path/configuration had been installed and verified; disabled scheduling alone did not prevent it. The earlier environment-variable blocker is withdrawn. Both nonblocking findings remain unchanged and outside this repair.
The repair uses the existing framework control, not a new guard or collector rewrite:
"on_demand_enabled": false
schedule.enabled also remains false. The schema preserves explicit false; EcsPipeline forwards it as onDemandEnabled; EcsStepFunction chooses CheckOnDemandEnabled, whose default is OnDemandDisabled before CreateRunRecord and EcsRunTask. Existing trusted SCHEDULED/EVENT handling and IAM are unchanged; there are no enabled schedules or declared upstream event triggers for this pipeline. The fix does not claim to prevent a privileged operator from deliberately bypassing normal invocation.
Only four files change in the repair: pipeline.json, README.md, OPERATIONS.md, and one five-line actual-config regression in tests/test_controller.py (+26/−6 total). Red→green: missing flag resolved to true and failed the regression; explicit false passes all 8 focused tests. The existing synthesized framework guard test also passes. Both independent reviewers passed the exact uncommitted diff (SHA-256 d4e04401fcb24f505f5386682543d7cfcb117590168039a51f765c5a97b22ad3) before the PR-context/commit gate; new-head remote CI and Mercy re-review are required after push. Old-head CI is not claimed as new-head acceptance.
Host entrypoint, interpreter, complete relative layout, root-owned configuration, hashes and real lease binding remain an explicit later activation gate. Keep both invocation controls disabled until those prerequisites are verified, then enable only the intended path through normal reviewed configuration and perform scoped live acceptance. No SSM-path edit, shared-framework change, source collection, AWS mutation, deployment or schedule activation is included in this repair. The existing deployed Raptor DSS reader remains unchanged.