<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
## 1. Summary
- Extends the existing taxonomy in scripts/issue_learning_loop.py (TAXONOMY_RULES) with four classes — doc-accuracy, silent-degradation, test-quality, style-nit — instead of building a second, parallel classifier (AI-191: one ingest, one key space).
- Adds scripts/finding_classification.py: classifies each drone-reviewer finding from its real inline-comment text (fetched from GitHub, not the review-summary body — see finding below), crosses it with the addresser's own reply-thread outcome (fixed/skipped/not-applicable/unreported/already-fixed), and persists one record per finding to reports/finding-classifications.json keyed by (repo, pr, comment_id) — re-running is a merge, not a recompute.
- Charts it weekly: scripts/eval_weekly_charts.py renders reports/finding-classification-trend.png (class volume over time + class × outcome), and src/eval-weekly.ts renders a ## Finding Classification section in the weekly markdown report.
- Backfilled a real baseline: 1,944 classified findings across 54 recently-merged PRs (49 trilogy-drones + 5 Klair).
## 2. Why It's Needed
We could measure *how many* reviewer findings there were and *how severe*, but not *what kind of problem* — so "what does our reviewer keep finding?" was answered by impression. That blocks three things: AI-213's reviewer-precision baseline, a proposed sixth "documentation" review dimension (~20% of review spend forever), and any learned-appendix prompt change (currently a guess about which lesson matters most). This ticket produces the measurement instrument those decisions need.
## 3. Changes
- scripts/issue_learning_loop.py — four new TaxonomyRule entries (doc-accuracy, silent-degradation, test-quality, style-nit), a TAXONOMY_CATEGORIES closed-union constant, and a cross-reference comment pointing at the new module for the outcome cross.
- scripts/finding_classification.py (new) — root-finding header parser (Sev · dim — the shape src/review-inline.ts posts per inline comment), addresser-reply parser (mirrors src/addresser.ts's renderReplyBody grammar), persisted ledger (load/merge/save), weekly summary/rollup, and a CLI (--pr, --backfill, --week-start/--week-end/--date-tag).
- scripts/eval_weekly_charts.py — load_classification_weekly_snapshots, draw_classification_trend, write_classification_trend_audit; wired into main(), non-fatal when the ledger is empty.
- src/eval-weekly.ts — ClassificationTrendResult/ClassificationTrendSummary types, loadClassificationTrend (injectable via a new WeeklyEvalOptions.loadClassificationTrend seam — kept separate from runPipelines so every existing test double for that option is untouched), renderClassificationTrendSection, wired into WeeklyEvalArtifact and runWeeklyEval.
- scripts/test_finding_classification.py (new, 38 tests) + src/eval-weekly.test.ts (8 new tests) — see Test plan.
- reports/ — the backfilled ledger (finding-classifications.json), one summary snapshot, the trend chart + audit.
- docs/decisions/ — one new entry.
A real bug found along the way (not fixed here — out of scope, flagged for a follow-up): issue_learning_loop.py's existing source_counts["drone"] breakdown classifies the review summary body. src/review-inline.ts (stripInlinedFindings) replaces any finding also posted inline with the literal placeholder "finding inlined as a PR comment." in that body — so a real share of that breakdown's historical other bucket is an artifact of classifying the placeholder, not genuine imprecision. finding_classification.py avoids this by reading the finding's real text from the inline-comment API directly (which the backfill needed anyway).
### Contract surface affected
None — classification is metadata only. It never touches a finding's severity, text, routing, or whether it gets addressed (verified: finding_classification.py only reads GitHub comments and writes its own separate ledger; nothing in src/addresser.ts / src/reviewer.ts was touched).
## 4. Breaking Changes
None. All new fields/files are additive:
- TAXONOMY_RULES gained four entries but zero findings moved between the original nine buckets on the 351-finding sample used to tune the new rules (measured — see Verification artifact); the new classes only pull findings out of other, they don't reclassify between old categories.
- WeeklyEvalArtifact gained one required field (classification_trend); all 7 pre-existing test fixtures updated in this PR.
- eval_weekly_charts.py's new chart/audit functions are additive and skip (not crash) when there's no classification data yet.
## 5. Test Plan
- [x] pnpm typecheck → clean.
- [x] pnpm exec vitest run → 3,529 tests passed (113 files), including 50 in src/eval-weekly.test.ts (8 new: loadClassificationTrend ×3, renderClassificationTrendSection ×4, one runWeeklyEval integration test).
- [x] node scripts/run-python-tests.mjs (pnpm test's Python leg) → 478 tests passed, including 38 new in scripts/test_finding_classification.py (closed-union assertions, header/reply parsers, merge idempotency, classification_sample_status threshold, summarize_records zero-fill + unclassifiable separation).
- [x] pnpm build → clean.
- [x] Live validation against real data (not just fixtures): ran python scripts/finding_classification.py --pr <owner>/<repo>#<n> ... against 54 real merged PRs via gh/GitHub API and inspected the persisted ledger + rendered chart — see Verification artifact.
- [ ] Operator-side: run python scripts/finding_classification.py --backfill --max-prs 50 against a fresh pr_data.json cohort periodically to keep the ledger current, and python scripts/eval_weekly_charts.py to regenerate the trend chart weekly (already wired into the normal eval-weekly flow going forward).
## 6. Verification Artifact
Real backfill distribution (1,944 findings, 54 merged PRs, reports/finding-classification-summary-2026-08-06.json):
| Category | Count | Share |
|---|---:|---:|
| error-propagation | 673 | 34.6% |
| other (unclassifiable) | 281 | 14.5% |
| edge-cases | 266 | 13.7% |
| input-validation | 195 | 10.0% |
| test-gaps | 160 | 8.2% |
| data-contract | 138 | 7.1% |
| state-concurrency | 66 | 3.4% |
| observability | 55 | 2.8% |
| doc-accuracy | 43 | 2.2% |
| security-authz | 22 | 1.1% |
| performance-cost | 16 | 0.8% |
| test-quality | 11 | 0.6% |
| style-nit | 11 | 0.6% |
| silent-degradation | 7 | 0.4% |
doc-accuracy is NOT the largest class — it's 9th of 13, at 2.2%. That's the headline this instrument was built to produce: it pushes back on the premise that a sixth "documentation" review dimension is obviously justified by volume. The unclassifiable bucket (14.5%) is disclosed as its own headline, not folded into a real class.
class × outcome (candidate FP signal — not adjudicated; a skip/not-applicable reason is the addresser's own opinion, AI-213 owns the human-adjudicated rate): overall fixed=1,508 (77.6%), skipped=85 (4.4%), unreported=72 (3.7%), not-applicable=17 (0.9%), already-fixed=1, no_addresser_reply=229 (11.8% — round hasn't happened / no reply thread found).
Old-vs-new distribution on the 351-finding tuning sample (trilogy-drones PRs #144–#156), confirming no cross-bucket drift among the original nine:
| Category | Old (9 rules) | New (13 rules) |
|---|---:|---:|
| error-propagation | 114 (32.5%) | 110 (31.3%) |
| other | 68 (19.4%) | 62 (17.7%) |
| edge-cases | 56 (16.0%) | 50 (14.2%) |
| input-validation | 33 (9.4%) | 31 (8.8%) |
| test-gaps | 27 (7.7%) | 26 (7.4%) |
| data-contract | 25 (7.1%) | 25 (7.1%) |
| observability | 18 (5.1%) | 18 (5.1%) |
| doc-accuracy | — | 13 (3.7%) |
| state-concurrency | 5 (1.4%) | 5 (1.4%) |
| security-authz | 4 (1.1%) | 4 (1.1%) |
| test-quality | — | 3 (0.9%) |
| silent-degradation | — | 2 (0.6%) |
| style-nit | — | 1 (0.3%) |
| performance-cost | 1 (0.3%) | 1 (0.3%) |
Zero findings reassigned between any two of the original nine buckets — every shift is extraction from other (or, for silent-degradation, a narrow proximity-gated slice of error-propagation's "silent/swallow" vocabulary) into a new class.
Rendered chart (reports/finding-classification-trend.png):
[Finding classification trend chart showing class volume over time and class x addresser outcome for the current week](https://cursor.com/agents/bc-cb26b63f-0fef-4683-93e2-c49640453449/artifacts?path=%2Fopt%2Fcursor%2Fartifacts%2Ffinding_classification_trend_chart.png)
TS mirror decision: none needed. scripts/finding_classification.py has exactly one runtime consumer — src/eval-weekly.ts reads its JSON summary opaquely (same as it already does for issue_learning_loop.py's output). This differs from src/human-review-findings.ts ↔ scripts/human_review_findings.py, which mirror parsing logic that both the TS ingest path (addresser.ts/runner.ts) and the Python report pipeline need independently at runtime — there is no TS runtime path here that needs to classify a finding into this taxonomy.
Scope note on the backfill: the 5/34 sampled Klair PRs that contributed records did so because a handful of comments happened to match the drone-reviewer header shape; the majority of sampled Klair PRs in this window carry only Mercy/human-formatted findings (different header shapes, intentionally out of scope for this module — see the docstring). The 1,944-finding baseline is therefore concentrated in trilogy-drones' own self-hosted review corpus; extending backfill coverage to more Klair PRs reviewed by the drone-reviewer itself is a natural follow-up once an operator has a fresh pr_data.json.
<sub>To show artifacts inline, <a href="https://cursor.com/dashboard/cloud-agents#team-pull-requests">enable</a> in settings.</sub>
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a href="https://cursor.com/agents/bc-cb26b63f-0fef-4683-93e2-c49640453449?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-cb26b63f-0fef-4683-93e2-c49640453449&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
