Skip to content

Commit 381c07f

Browse files
chipiclaude
andcommitted
bugfix-fleet(triage-v2): intent-source gate — poison path closed, recall cost measured
agents/triage.md v2: every acceptance criterion must cite an intent_source (reporter|spec|repo-data|code-invariant|baseline); own-analysis is not a source; code-invariant is an acceptance floor; kick-back re-audits sources before sharpening. triage_run.sh: v2 object schema in shape check (uncited counter), prompt_ver stamped in the ledger, shape.txt persisted. orchestrate.sh: deterministic downgrade — actionable + uncited criteria = needs-info, never dispatched. A/B on the same 5 L0s (BAKEOFF 6.3 session 3d): intake strictly better — mission-arc needs-info with the exact intent question (v1: 3 confident inventions), fly-physics ignored the schema and the mechanical gate caught it. End-to-end ships less (1 vs 3): v1's credits and look-angles wins leaned on correct-invention and smuggle, removed by design; both v2 worker FAILs n=1, mechanism verified by ticket diffs. Caveat on file: needs-info is terminal only because the replay eval has no reporter — reporter-oracle is the next instrument piece; composed dispositions converge with SIGNALS.md 7.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WThYGP2nTpVk4c44jcbBoZ
1 parent ed655bd commit 381c07f

8 files changed

Lines changed: 128 additions & 23 deletions

File tree

bugfix-fleet/BAKEOFF.md

Lines changed: 48 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -610,11 +610,57 @@ Readings:
610610
expensive ones — one more argument for the intent-source gate firing
611611
*before* the first specialist episode.
612612

613+
**Observed (2026-07-24, session 3d): triager v2 A/B — the intent gate
614+
works at intake and costs recall end-to-end.** `agents/triage.md` v2 adds
615+
the intent-source schema (every acceptance criterion cites
616+
`reporter | spec | repo-data | code-invariant | baseline`; own-analysis is
617+
not a source; code-invariant is a floor). Ledger rows stamped
618+
`prompt_ver`; orchestrator now **deterministically downgrades** an
619+
actionable verdict with uncited criteria to needs-info (gates on
620+
`shape.txt`, no LLM). Same 5 L0s, same flash triager:
621+
622+
| ticket | v1 verdict → outcome | v2 verdict → outcome |
623+
|---|---|---|
624+
| mission-arc | actionable (invented ×3) → stuck, ~$0.25 | **needs-info**, asks the exact intent question ✓ |
625+
| fly-physics | actionable (invented ×3) → stuck | actionable but **4 uncited → auto-downgrade to needs-info** ✓ (model ignored the schema; the mechanical gate caught it) |
626+
| credits | actionable → **shipped** | actionable, cited → worker **FAIL** (n=1) |
627+
| look-angles | actionable (+smuggle) → **shipped** | actionable, clean → worker **FAIL** (n=1) |
628+
| 335 (control) | actionable → shipped | actionable, cited → **shipped**|
629+
630+
Readings — both directions, honestly:
631+
- **Intake-level: v2 strictly better.** The needs-info valve fired
632+
correctly (0/11 calls in v1 → fired on mission-arc in v2); where prompt
633+
discipline failed (fly-physics emitted plain uncited strings), the
634+
schema check caught it mechanically — defense in depth as designed. The
635+
poison-ticket path (worst measured cost, 782s/231k grinds) is closed.
636+
- **End-to-end vs the binary oracle: v2 ships less (1 vs 3).** Mechanism,
637+
verified by diffing v1/v2 tickets: v1's credits win leaned on **correct
638+
invention** (CMSA→cnsa is a maintainer judgment; v1 guessed it right),
639+
and v1's look-angles win leaned on **smuggled localization with
640+
operationally sharp criteria**. v2 refuses both: credits narrowed to the
641+
citable CSA slice and explicitly deferred CMSA/USAF/SpaceIL as "needs a
642+
design decision" (epistemically correct!); look-angles swapped sharp
643+
ECI-value checks for abstract invariants that mostly already held. Both
644+
worker FAILs are n=1 but the mechanism is structural: **the gate trades
645+
lucky-invention recall for zero-poison precision.**
646+
- **The eval instrument now under-scores v2 by design:** needs-info is a
647+
*terminal* state in the replay eval because there is no reporter to
648+
answer. In the real fleet, needs-info returns to a reporter and the
649+
loop continues. Next instrument gap: a **reporter-oracle** (answers
650+
needs-info questions from the golden fix's knowledge) so the full loop
651+
can be scored. Until then, v1-vs-v2 end-to-end counts are not
652+
comparable at face value.
653+
- **Convergence with the signal-fleet design:** credits wants a **composed
654+
disposition** — file the citable slice AND escalate the judgment slice —
655+
exactly SIGNALS.md §7.1. The binary File/reject shape loses value here.
656+
613657
**Two scores, kept separate:**
614658
- **Active-triage (intake) score** — L0 garbage → L1-or-correctly-rejected: did it
615659
produce a solvable problem, correctly reject the unsolvable, classify
616-
given/derivable/missing right? *(First measurement: the session-3c block
617-
above — 2 shipped, 2 invented-instead-of-needs-info, 1 control clean.)*
660+
given/derivable/missing right? *(Measured: session-3c block — v1: 2
661+
shipped, 2 invented-instead-of-needs-info, 1 control clean; session-3d —
662+
v2 closes the invention path, at a recall cost pending a
663+
reporter-oracle.)*
618664
- **Harness (fix) score** — L1 → correct fix via *its own* recon (§7), plus the
619665
**min upping-level to pass**.
620666

bugfix-fleet/agents/triage.md

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ name: triage
33
description: Active triager — normalizes a raw bug issue into a fix-ready L1 problem or rejects it. First pass on every `bug`-labeled issue; re-entered on harness kick-back with the failed attempt as evidence. Never fixes, never localizes on first pass.
44
model: deepseek/deepseek-v4-flash
55
area: intake
6+
version: 2
67
---
78

89
# active triager
@@ -29,10 +30,35 @@ You never write the fix and you never prescribe one.
2930
HUD", not "the vis-viva function"). Measured reason: a name-trap in the
3031
ticket beats any doc substrate ~2/3 of the time (BAKEOFF §6.3 fixmap);
3132
a wrong name you pass through poisons the whole attempt.
32-
4. **Gate.** The test is razor-sharp: **actionable ⟺ acceptance criteria are
33-
statable ⟺ a pass/fail oracle could exist.** If you cannot state what
34-
"fixed" means concretely enough that a test could decide it, do not fake
35-
it — return `needs-info` (say exactly what is missing) or `reject`.
33+
4. **Ground every acceptance criterion in an intent source.** Correlating
34+
evidence proves what the code *does*; it can never prove what the code
35+
*should* do. Each acceptance criterion must carry an `intent_source`:
36+
- `reporter` — the report states it (quote it in `source_ref`);
37+
- `spec` — a spec/ADR/PRD/doc states it (name it);
38+
- `repo-data` — a data file, constant, or registry you actually read
39+
states it (name the file);
40+
- `code-invariant` — self-evident: a crash, NaN, 500, null-deref — the
41+
app must not do this, no document needed. **This is an acceptance
42+
FLOOR, not the acceptance**: "the NaN is gone" does not say what the
43+
value should BE. If the fix requires choosing among coherent
44+
alternatives (a fallback behavior, a formula, a mapping) and only
45+
code-invariant sources exist, the choice is the maintainer's —
46+
`needs-info`.
47+
- `baseline` — a prior measured window defines normal (state it).
48+
49+
**"Derived from my own analysis of the code" is NOT a source.** If you
50+
catch yourself writing a criterion you cannot attribute to one of the
51+
five sources, that criterion is an invention — delete it, and if the
52+
problem cannot stand without it, the verdict is `needs-info`, naming
53+
exactly the question the reporter must answer. Being unable to state
54+
acceptance is a strong, correct verdict — an invented acceptance is
55+
worse than none: it sends the specialist to satisfy the wrong contract
56+
at full cost.
57+
5. **Gate.** The test is razor-sharp: **actionable ⟺ acceptance criteria are
58+
statable from citable intent ⟺ a pass/fail oracle could exist.** If you
59+
cannot state what "fixed" means concretely enough that a test could
60+
decide it — with sources — do not fake it: `needs-info` (say exactly
61+
what is missing) or `reject`.
3662

3763
### L1 boundaries (hard)
3864

@@ -63,9 +89,12 @@ verdict × scope 2×2 (BAKEOFF §6.2, measured at k=3):
6389
- **FAIL + scope=yes (acceptance gap — right place, wrong "done"):**
6490
the specialist found the file but ground without a target (measured
6591
signature: 3–8× token burn, right file, oracle still red). Your acceptance
66-
criteria were not decisive. Sharpen them into concrete observable
67-
outcomes (exact values, exact behaviors) or, if the reporter never said
68-
and nothing in repo/docs implies it, downgrade to `needs-info`.
92+
criteria were not decisive — or were invented. Re-audit every criterion's
93+
`intent_source` first: a criterion you cannot attribute is the likely
94+
poison, and replacing one invention with a sharper invention repeats the
95+
failure at full cost (measured: three invented theories in three rounds,
96+
all FAIL). Sharpen only from citable sources; otherwise downgrade to
97+
`needs-info` naming the exact question.
6998
- **Corroboration, free:** cheap-and-fast FAIL smells of wrong-place;
7099
expensive-grind FAIL smells of no-definition-of-done. Use it to
71100
sanity-check the scope signal, not to override it.
@@ -84,7 +113,11 @@ The orchestrator consumes this verbatim; prose outside the JSON is dropped.
84113
"problem": {
85114
"symptom": "what is observably wrong, in behavior/owner terms",
86115
"expected": "what correct behavior looks like",
87-
"acceptance": ["concrete pass/fail criteria — each one testable"],
116+
"acceptance": [
117+
{ "criterion": "concrete pass/fail statement",
118+
"intent_source": "reporter | spec | repo-data | code-invariant | baseline",
119+
"source_ref": "the quote / file / doc / window that states it" }
120+
],
88121
"evidence": "repro steps, logs, values from the report",
89122
"area": "backend | database | ui | docs | infra",
90123
"domain_facts": ["facts the repo cannot supply; [] if none"],

bugfix-fleet/bakeoff/bugs/triaged/orrery-335-mission-event-merge-L0-triaged.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,5 @@
99
"area": "data-merge",
1010
"difficulty": "medium",
1111
"level": "L1-triaged",
12-
"description": "Cassini's 4 interplanetary flyby events (Venus #1, Venus #2, Earth, Jupiter) show as 'FLYBY' with empty notes in the CAPCOM event ticker on /fly, instead of their proper names and descriptions from the mission flight data.\n\nExpected: The CAPCOM ticker should show each flyby with its per-event label (e.g. 'Venus #1 — gravity assist') and description (e.g. 'First of four assists building the velocity for Saturn arrival. Closest approach 284 km above the surface.') from the structural flight event, not the generic type default 'FLYBY'.\n\nAcceptance criteria:\n- mergeFlightEvents output includes Cassini's 4 structural flyby events with their `label` values as the event label, not 'FLYBY'\n- Cassini's structural flyby events with a non-empty `description` have that description surfaced as the event note, not ''\n- Collision dedup with editorial events within 0.05-day tolerance still applies — editorial VENUS GRAVITY ASSIST (met 207) does NOT suppress the Venus #1 flyby (met 193, 14 days away)\n- Structural events without explicit `label` or `description` fields still fall back to the current type-keyed defaults (flyby → 'FLYBY' / '', etc.)\n- The same propagation works for any structural flight event that has `label` set (Galileo, Voyager 2, New Horizons, etc.)"
12+
"description": "On the /fly CAPCOM event ticker, structural flyby events from missions with multi-flyby trajectories show only the generic type label 'FLYBY' with an empty note, instead of the per-event labels and descriptions authored in the mission data. Affects at minimum Cassini, MESSENGER, BepiColombo, Voyager 2, New Horizons, Pioneer 11, Ulysses, Hayabusa2, Vega-1, Vega-2, Artemis 2, Apollo 13, and any other mission with label-bearing `flight.events[].label` fields and no editorial overlay event at the same MET.\n\nExpected: The CAPCOM ticker shows the per-event `label` (e.g. 'Venus #1 — gravity assist') and `description` (e.g. 'First of four assists…') from each structural flight event, falling back to the type-based default only when the per-event fields are absent.\n\nAcceptance criteria:\n- Cassini's flyby events appear in the CAPCOM ticker with the labels from `flight.events[].label`: 'Venus #1 — gravity assist', 'Venus #2 — gravity assist', 'Earth — gravity assist', 'Jupiter — gravity assist', not 'FLYBY'. [reporter: The report says 'The mission data definitely has proper names for these events.']\n- Cassini's flyby events appear in the CAPCOM ticker with the notes from `flight.events[].description`, not empty strings. [reporter: The report says 'every flyby in the event ticker ... [has] an empty note' and the data has `description` fields for each.]\n- For every mission with `flight.events[].label` set, the CAPCOM ticker renders that label instead of the generic type default, and falls back to the type default only when `label` is absent. [repo-data: `static/data/schemas/mission.schema.json` line 344: 'Editorial chip label shown over the /fly milestone overlay. Falls back to a humanised event-type when omitted.' — the schema explicitly says fallback is only when omitted, meaning present labels must be used.]\n- The CAPCOM ticker note field for a structural event shows `description` when present, and falls back to the type default's note string when absent. [repo-data: `static/data/schemas/mission.schema.json` line 350: 'Editorial 1–2 sentence flavour text shown when the milestone is active. Optional; the /fly chip collapses to label-only when absent.']"
1313
}

bugfix-fleet/bakeoff/bugs/triaged/orrery-credits-L0-triaged.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,5 @@
99
"area": "data-mapping",
1010
"difficulty": "easy-medium",
1111
"level": "L1-triaged",
12-
"description": "On the /credits bill-of-materials page, images whose provenance records identify a specific space agency (CMSA, USAF, CSA, SpaceIL) are appearing under the \"Wikimedia Commons contributors\" section instead of under that agency's own section header.\n\nExpected: Every image on /credits should be grouped under the section of the agency that produced it — the upstream publisher — regardless of the conduit (Wikimedia Commons, NASA Images API, etc.) through which Orrery downloaded the file. When the image-provenance entry's `agency` field names a known space agency (e.g. CMSA → China National Space Administration, CSA → Canadian Space Agency, USAF → U.S. Air Force / Space Force, SpaceIL), the image must appear under that agency's section, not under Wikimedia Commons.\n\nAcceptance criteria:\n- An image whose `agency` field is `CMSA` must appear under a section whose source ID is `cnsa` (or a dedicated `cmsa` section) on /credits — not under `wikimedia-commons`.\n- An image whose `agency` field begins with `CSA` (first token before \" / \") must appear under source ID `csa` on /credits.\n- An image whose `agency` field is `USAF` must appear under source ID `us-space-force` (or a dedicated `usaf` section) on /credits — not under `wikimedia-commons`.\n- After the fix, the `/credits` page must show ≤ 5 image bundles (accounts for generic labels like `publisher`, `Multi-agency`, `public-domain`) under the Wikimedia Commons section that have a known agency name — currently 153 such bundles (121 CMSA + 20 USAF + 9 SpaceIL + 2 CSA + 1 CFHT/EHT) would move to correct sections, leaving only 62 generic-label bundles in Commons.\n- The `/credits` page must still show the correct per-image TASL row including the `source_url` pointing to the Wikimedia file page, even when the image's section is the producing agency rather than Commons.\n\nDomain facts:\n- The source-logos registry at `static/data/source-logos.json` already defines source entries for: cnsa (CNSA), csa (Canadian Space Agency, with logo), us-space-force (U.S. Space Force), and 43 other sources — but not for CMSA (China Manned Space Agency), USAF (U.S. Air Force), or SpaceIL\n- CMSA (China Manned Space Agency) is the operator of China's human spaceflight program (Shenzhou, Tiangong) and is closely affiliated with CNSA; Chang'e lunar missions are operated by CNSA but CMSA is sometimes credited as author on mission imagery uploaded to Commons\n- USAF (United States Air Force) produced the X-37B OTV mission photos; the X-37B program transferred to the U.S. Space Force (USSF) in 2020. USAF works are U.S. Government public domain (17 U.S.C. § 105), same as US Space Force imagery\n- SpaceIL is an Israeli nonprofit that built and operated the Beresheet lunar lander; no existing source-logos entry covers it\n- ADR-046 primary-credit rule: the first agency token (split on \" / \") wins for attribution"
12+
"description": "On /credits, the Canadian Space Agency (CSA) images for Canadarm2 appear under the 'Wikimedia Commons contributors' section instead of the 'Canadian Space Agency' section. Images of other space agencies (CMSA — Chinese Manned Space Agency, USAF, SpaceIL) also fall through to Wikimedia Commons, but those lack source-logos entries and need a design decision.\n\nExpected: Images sourced from or depicting hardware of a space agency that has a source-logos entry on /credits must appear under that agency's section. Specifically, entries with agency 'CSA / NASA' must appear under the 'Canadian Space Agency' (source_id: 'csa') section, not under Wikimedia Commons.\n\nAcceptance criteria:\n- An image-provenance entry with agency='CSA / NASA' (primary token 'CSA') must route to source_id='csa' in provenanceSourceId(). [repo-data: static/data/source-logos.json entry: { \"id\": \"csa\", \"name\": \"Canadian Space Agency\", \"kind\": \"space-agency\" }]\n- The 2 image-provenance entries with path matching /images/iss-modules/canadarm2/* must appear under the 'csa' section on /credits, not under 'wikimedia-commons'. [repo-data: static/data/image-provenance.json entries for canadarm2 (agency='CSA / NASA')]\n- provenanceSourceId() must not return 'wikimedia-commons' for any entry where the primary agency token has a matching source-logos entry. [code-invariant: The /credits page groups by upstream publisher per editorial rule documented in src/lib/credits-grouping.ts §'Editorial rule for the public credits page']"
1313
}

0 commit comments

Comments
 (0)