SimplicityGuy
diff --git a/‎.planning/PROJECT.md‎
Lines changed: 2 additions & 0 deletions b/‎.planning/PROJECT.md‎
Lines changed: 2 additions & 0 deletions
diff --git a/‎.planning/REQUIREMENTS.md‎
Lines changed: 12 additions & 12 deletions b/‎.planning/REQUIREMENTS.md‎
Lines changed: 12 additions & 12 deletions
@@ -112,6 +112,8 @@ Get 200K messy music and concert files properly named, organized into logical fo
 
 **Phase-by-phase execution history (2026.7.5):**
 
+**2026.7.5 Parallel Enrich DAG — Phase 87 (Operator UI — Stage Matrix, Failure Retry, Eligibility Trace & Priority) COMPLETE + verified 2026-07-11 (5/5 must-haves).** Replaces the raw-enum "State" column with a derived per-file stage matrix and gives the operator failure visibility+retry, a "why not eligible?" trace, force-skip, an orphaned-work count, and the restored per-stage priority stepper (UI-01..05, PRIO-01). Shipped across 9 plans / 6 waves + a gap-closure plan: 87-01 the `stage_skip` `(file_id,stage)` sidecar (migration `037`); 87-02 threaded a `skipped` marker (`Status.SKIPPED` + `skipped_clause`) through the single-source derivation with the precedence `in_flight ≻ done ≻ skipped ≻ failed ≻ not_started`; 87-03 locked the marker against SQL⇔Python drift; 87-04 the 5-bucket pill + 6-pill matrix + paginated, per-row-derived, degrade-safe `GET /pipeline/files` (no whole-corpus scan; all four Phase-77 partial indexes back the probes); 87-05 the URL-carried status/failure filter lens + retired the raw-enum column; 87-06 the single-row eligibility trace (honest — a skipped upstream renders as a gating blocker deferred to Phase 90, not "satisfied") + the enrich-only, sanitized, additive force-skip writer; 87-07 per-file + bulk retry affordances (analyze stays manual, terminal-guard preserved; fingerprint self-retries); 87-08 the recovery-parity orphan-count rail badge + the priority/pause/resume re-wire. **Six issues found and fixed inline during execution** (per the never-skip rule), each with a mutation-verified regression guard: a pre-existing `psycopg2` DSN footgun in `tests/conftest.py` (bare `postgresql://` → sync dialect; `_coerce_async_dsn`); a **behavior-5 recovery gap** where `recover_orphaned_work` re-enqueued force-SKIPPED fingerprint files (`reenqueue.py` `fingerprint_done` now `or_(done_clause, skipped_clause)`); two post-merge guard failures (table registry missing `stage_skip`; dead-template guard false-trip on a comment); and code-review **CR-01** (force-skip 500 on duplicate → `on_conflict_do_nothing`) + **WR-01** (validate the reason after sanitize). An app-wide `[x-cloak]` CSS rule (Alpine never injected it) was also fixed. **UI-01/UI-02 initially verified as a gap** (3/5) — the derived files matrix was built+tested but had no shell nav entry — closed by gap-closure plan 87-09 (a first-class "Files" rail node → `/s/files` with the inherited fragment/full-page fork), re-verified 5/5. Backlogged with a 2026.7.5 close-out gate: **WR-02** (the orphan-count badge runs recovery's full ledger derivation on every 5s poll — a read-path perf bound). Ships on branch `SimplicityGuy/phase-87`. **Next: Phase 88 (Lane / Agent Drill-In).**
+
 **2026.7.5 Parallel Enrich DAG — Phase 81 (Per-Stage Failure Persistence & Retry Paths) COMPLETE + verified 2026-07-09 (4/4 success criteria).** Every enrich stage now has a durable failure marker and a retry story, closing the latent bug where a failed metadata extraction recorded *nothing* and became invisible-and-permanently-ineligible. Shipped across 6 plans / 2 waves: (1) `enums/stage.py` gains the two orthogonal DB-free tables `FAILURE_IS_TERMINAL` {analyze:True, metadata:True, fingerprint:False} and `ELIGIBLE_AFTER_FAILURE` {analyze:False, metadata:True, fingerprint:True} — `FAILURE_IS_TERMINAL` was cited by 80-CONTEXT D-01 but had existed in no `.py` until now — plus the pure `domain_completed()` and its SQL twin `domain_completed_clause()` in `services/stage_status.py`, drift-locked by the extended Phase-78 equivalence matrix. Both twins are deliberately **ledger-agnostic** (they do not model `in_flight`); Phase 80's recovery callers must apply the ledger check separately. (2) Migration `033` runs a mixed-row cleanup UPDATE (done wins — `failed_at` cleared, `analysis_completed_at` retained, D-09/D-04) BEFORE `create_check_constraint`, since `032`'s unguarded backfill had already produced mixed rows in the live corpus; the `analysis_completed_at` XOR `failed_at` CHECK is mirrored into the ORM `__table_args__` so autogenerate still yields an empty diff. The *destructive* Phase-90 migration is renumbered `033 → 034` across all forward-looking planning docs (D-08); dated historical records consciously keep `033`. (3) FAIL-02: `report_metadata_failed` persists a `metadata` row (`failed_at` + `error_message`) via `pg_insert(...).on_conflict_do_update` instead of writing nothing; `put_metadata` clears the marker on success; the request body is **optional** (`MetadataFailurePayload | None`, no `Body()` wrapper) so a bodyless POST from an older agent still 200s and clears the ledger (CR-02 version-skew guard, D-10). (4) FAIL-01: `report_analysis_failed` dual-writes `analysis.failed_at` + `error_message` alongside the still-live `state = ANALYSIS_FAILED`, clearing `analysis_completed_at` to satisfy the new CHECK; `put_analysis` clears the marker on success. The SC#1 backfill itself lives in migration `032` (Phase 77, `faee8b8a`) — SC#1 holds on the 77+81 combination, verified explicitly rather than assumed. (5) FAIL-03: `POST /pipeline/metadata-failed/retry` bulk operator retry reuses `failed_clause(Stage.METADATA)` and mirrors `retry_analysis_failed`'s Phase-30 guard (resolve the per-agent `meta` lane once, catch `NoActiveAgentError`, never fall through to the consumer-less default queue); per D-11 it leaves the failure row in place — only a successful re-run clears it. (6) FAIL-04 ships **no new writer** by design: characterization tests prove `report_fingerprint_failed` persists no `fingerprint_results` row (and writes no `engine='_task'` sentinel that would poison the per-engine badge joins); the durable, auto-retryable marker is the existing `status='failed'` row from `put_fingerprint`, and a failed fingerprint stays eligible (ELIG-04). Verifier 4/4 (independent codebase read + its own test re-runs). Post-merge gates: mypy clean (203 files), all nine buckets green (3,176 tests), `just docs-drift` green. Code-review gate: 2 blockers + 6 warnings + 4 info — **both blockers FIXED before phase close**, each proven non-vacuous by reverting the fix and watching the new tests go red: CR-01 (`1ff92265`) the pre-existing `retry_analysis_failed` flipped only `files.state` and left `analysis.failed_at` set, stranding every retried file as `fingerprinted`-yet-`failed` — a row the Phase 79 shadow gate reads as divergent and `domain_completed(ANALYZE)` reads as "un-processable", so Phase 80 recovery would have skipped exactly the files the operator just re-enqueued; CR-02 (`a6398d33`) `domain_completed_clause` raised a bare `KeyError` on the four downstream stages while the Python twin short-circuited on `DONE` and returned `True` — a silent twin divergence on every non-failed downstream row, now a symmetric fail-loud `ValueError` matching `reenqueue.py`'s existing "live-keys-only … no domain predicate" classification. Two warnings remain OPEN in `deferred-items.md`: **WR-02** — the `domain_completed` drift-lock excludes the `in_flight ∧ failed` seeds, a cell FAIL-03's retry now *routinely* produces (Phase 80 must add an `inflight` term to the SQL twin or gate at the call site); **WR-01** — the metadata failure upsert can mark a row that already carries real tags as failed. Ships as its own PR on a worktree branch. **Next: Phase 82 (Counts & Pending-Set Cutover).**
 
 **2026.7.5 Parallel Enrich DAG — Phase 79 (Shadow-Compare Gate, live corpus) COMPLETE + verified 2026-07-08 (11/11 code-verifiable truths; 1 human item deferred per D-02).** The standing state↔derived implication gate that phases 80–90 keep green and the hard precondition for the destructive `033` (Phase 90). Shipped across 2 plans / 2 waves over the ONE shared assertion core (D-01): (1) `src/phaze/services/shadow_compare.py` — an `INVARIANTS` registry with one `state ⇒ derived-condition` implication per `FileState` value (16 entries; `DISCOVERED` documented-vacuous), `Invariant`/`InvariantResult`/`Report` dataclasses, and async `run_shadow_compare` running a per-invariant corpus-wide anti-join (`state=X AND NOT <derived>`) returning counts + capped `file_id` samples + `hard_fail_total`; reuses Phase-78 `done_clause`/`failed_clause` (D-03, never `stage_status_case`), asserts implication-not-equality, PUSHING/PUSHED loosen to `cloud_job` row-existence (RESEARCH A3/OQ1) while AWAITING_CLOUD keeps `status='awaiting'`, apply-outcome states assert `proposals.status` (not `execution_log`), and the `{fingerprinted, local_analyzing}` soft allowlist is counted-but-never-gated (D-06). (2) `src/phaze/cli/shadow_compare.py` — a thin `python -m phaze.cli.shadow_compare` argparse runner + `[group('db')] shadow-compare *ARGS` justfile recipe driving the SAME core (no duplicated logic), exit 1 iff any HARD invariant diverges (D-05), `--sample-cap`/`--verbose`/`--database-url` flags, DSN never leaked. Hermetic fixture-corpus gate `tests/integration/test_shadow_compare.py` — non-vacuous divergent RED cell + consistent GREEN cell per HARD invariant, implication cell, soft-allowlist cell, DB-free registry cell locking D-04 coverage + D-06 allowlist, and CLI-exit cells; `just test-bucket integration` green in isolation (130 passed, 100% core coverage). Verifier 11/11 (independent codebase read + its own 130/130 suite re-run). Code-review gate: 1 blocker + 2 warnings + 1 info — ALL fixed inline (`ba489b62`): CR-01 (a data-loss footgun — the CLI cells' committed `TRUNCATE agents CASCADE` would wipe the DEV DB on a bare `pytest` with `TEST_DATABASE_URL` unset; now a module-level `_test`-DB guard refuses any non-`_test` target), WR-01 (DSN parse-error redaction + password-masking `URL` threaded to the engine), WR-02 (`--sample-cap` rejects negatives), IN-01 (`ORDER BY id` for deterministic samples). One human-verification item deferred by design: the live ~200K-corpus run (SC-3) is explicitly held to the next homelab rollout per D-02, tracked as the sole Manual-Only check in `79-VALIDATION.md`/`79-HUMAN-UAT.md` and must be recorded before Phase 90's `033`. Ships as its own PR on a worktree branch. **Next: Phase 80 (Recovery / Re-enqueue Cutover) — the first reader cut over off `FileRecord.state`, deliberately before the pending-set/counts readers (READ-03).**
 
@@ -57,11 +57,11 @@
 
 ### Operator UI (UI)
 
-- [ ] **UI-01**: The file-row "State" display is derived from `stage_status` per stage (a per-file stage matrix), replacing the raw-enum-string column; the list is paginated and never renders a query that scans the whole 200K corpus per poll.
-- [ ] **UI-02**: The operator can see failed files per enrich stage and trigger a retry from the console (fingerprint/metadata retry + the existing manual analyze retry). *(failure visibility + retry affordance)*
-- [ ] **UI-03**: For any file not in a stage's pending set, the operator can see **why it is not eligible** — an eligibility trace over the pure `eligible()` conjuncts (done? in-flight? upstream unmet? terminally failed?). *(closes gap G-04; the diagnostic whose absence hid the current deadlock)*
-- [ ] **UI-04**: The operator can force a stage to done / skip a stage for a specific file, so the `failed` bucket for genuinely-unprocessable files can converge rather than accumulate permanently. *(gap G-03)*
-- [ ] **UI-05**: An orphaned/stuck-work count is surfaced (files with an in-flight marker but no progress), derived for free from the chosen `in_flight` source.
+- [x] **UI-01**: The file-row "State" display is derived from `stage_status` per stage (a per-file stage matrix), replacing the raw-enum-string column; the list is paginated and never renders a query that scans the whole 200K corpus per poll.
+- [x] **UI-02**: The operator can see failed files per enrich stage and trigger a retry from the console (fingerprint/metadata retry + the existing manual analyze retry). *(failure visibility + retry affordance)*
+- [x] **UI-03**: For any file not in a stage's pending set, the operator can see **why it is not eligible** — an eligibility trace over the pure `eligible()` conjuncts (done? in-flight? upstream unmet? terminally failed?). *(closes gap G-04; the diagnostic whose absence hid the current deadlock)*
+- [x] **UI-04**: The operator can force a stage to done / skip a stage for a specific file, so the `failed` bucket for genuinely-unprocessable files can converge rather than accumulate permanently. *(gap G-03)*
+- [x] **UI-05**: An orphaned/stuck-work count is surfaced (files with an in-flight marker but no progress), derived for free from the chosen `in_flight` source.
 
 ### Performance (PERF)
 
@@ -80,7 +80,7 @@
 
 *Folded in from the #222 post-deploy backlog. Backend is live end-to-end; only the v7.0-deleted UI control is missing.*
 
-- [ ] **PRIO-01**: The operator can change a per-stage job priority from the shell (a stepper wired to the existing `POST /pipeline/stages/{stage}/priority` endpoint — ▲ raises priority / lowers the number), re-connecting the orphaned setter; pause/resume controls are surfaced too if that endpoint is likewise orphaned.
+- [x] **PRIO-01**: The operator can change a per-stage job priority from the shell (a stepper wired to the existing `POST /pipeline/stages/{stage}/priority` endpoint — ▲ raises priority / lowers the number), re-connecting the orphaned setter; pause/resume controls are surfaced too if that endpoint is likewise orphaned.
 
 ### Lane / Agent Drill-In (DRILL)
 
@@ -149,11 +149,11 @@
 | SIDECAR-01 | Phase 83 | Complete |
 | SIDECAR-02 | Phase 84 | Complete |
 | SIDECAR-03 | Phase 86 | Complete |
-| UI-01 | Phase 87 | Pending |
-| UI-02 | Phase 87 | Pending |
-| UI-03 | Phase 87 | Pending |
-| UI-04 | Phase 87 | Pending |
-| UI-05 | Phase 87 | Pending |
+| UI-01 | Phase 87 | Complete |
+| UI-02 | Phase 87 | Complete |
+| UI-03 | Phase 87 | Complete |
+| UI-04 | Phase 87 | Complete |
+| UI-05 | Phase 87 | Complete |
 | PERF-01 | Phase 77 | Complete |
 | PERF-02 | Phase 82 | Complete |
 | MIG-01 | Phase 77 | Complete |
@@ -163,7 +163,7 @@
 | LEGACY-01 | Phase 89 | Pending |
 | LEGACY-02 | Phase 89 | Pending |
 | LEGACY-03 | Phase 89 | Pending |
-| PRIO-01 | Phase 87 | Pending |
+| PRIO-01 | Phase 87 | Complete |
 | DRILL-01 | Phase 88 | Pending |
 | DRILL-02 | Phase 88 | Pending |
 | DRILL-03 | Phase 88 | Pending |