Phase 78: Derivation Layer, Eligibility & Anti-Drift Test Harness

.planning/PROJECT.md

@@ -106,7 +106,11 @@ Get 200K messy music and concert files properly named, organized into logical fo
 
 **2026.7.2 Multi-Compute Agents — SHIPPED + milestone COMPLETED 2026-07-06.** All 5 phases (72–76) merged (PRs #209/#210/#211/#213/#214). Milestone audit PASSED — 15/15 requirements, 5/5 phases verified & Nyquist-compliant, 6/6 cross-phase integration seams wired, E2E N-compute dispatch→push→reconcile→render flow complete. N cloud-compute agents now dispatch/route/reconcile/fail-isolate simultaneously (compute-side twin of the N-Kueue support) with the `≤1-compute` fail-fasts retired and zero new dependencies. Four low-severity/cosmetic review items closed at close-out by quick `260706-odc` (116 tests green). Deferred to v2: PROV-01 (N-compute-aware orphan recovery), PROV-02/03. Archived to `milestones/2026.7.2-{ROADMAP,REQUIREMENTS,MILESTONE-AUDIT}.md`; `REQUIREMENTS.md` removed (fresh for next milestone). Codebase: ~28.4k Python LOC in `src/`. Remaining ship step: push the `2026.7.1` tag (per STATE.md).
 
-**Active milestone: 2026.7.5 Parallel Enrich DAG (started 2026-07-08).** Retire the linear `FileState` enum; derive per-file, per-stage status from output tables so metadata/fingerprint/analyze are genuinely parallel. Fresh branch `SimplicityGuy/true-parallel` off `main` @ `ce0c6434` (includes PR #221). Phases continue from 76 (start at 77). Requirements + roadmap being defined this cycle.
+**Active milestone: 2026.7.5 Parallel Enrich DAG (started 2026-07-08).** Retire the linear `FileState` enum; derive per-file, per-stage status from output tables so metadata/fingerprint/analyze are genuinely parallel. Fresh branch `SimplicityGuy/true-parallel` off `main` @ `ce0c6434` (includes PR #221). 14 phases (77–90) mapped; requirements + roadmap defined. **Phase 77 (additive schema `032` + rescan-wipe fix) shipped — PR #223. Phase 78 (Derivation Layer, Eligibility & Anti-Drift Test Harness) COMPLETE + verified 2026-07-08.** Next: Phase 79 (Shadow-Compare Gate, live corpus).
+
+**Phase-by-phase execution history (2026.7.5):**
+
+**2026.7.5 Parallel Enrich DAG — Phase 78 (Derivation Layer, Eligibility & Anti-Drift Test Harness) COMPLETE + verified 2026-07-08 (12/12 must-haves).** The single-source-of-truth predicate layer shipped as two new modules, purely additive (no reader/writer cut over — that is Phase 79+). Shipped across 2 plans / 2 waves: (1) `src/phaze/enums/stage.py` — the DB-free, agent-safe half: `Stage`/`Status` StrEnums, `ELIGIBILITY_DAG` topology, the pure-Python `resolve_status()` precedence ladder (`in_flight ≻ done ≻ failed ≻ not_started`, encoding DERIV-02/03/05 + D-03 metadata-failure-only-is-FAILED), and the pure `eligible()` predicate (ELIG-01 enrich independence, ELIG-02 apply-gated-on-approved-proposal, ELIG-03 terminal-failed-analyze guarding the 44.5K over-enqueue class, ELIG-04 failed-fingerprint-stays-eligible); a subprocess banned-import test enforces the T-78-01 agent boundary (no `phaze.models`/`phaze.database`/`sqlalchemy`), 100% module coverage. (2) `src/phaze/services/stage_status.py` — the SQLAlchemy `ColumnElement[bool]` twin: per-stage `done_clause`/`failed_clause`, `stage_status_case` CASE ladder, `inflight_clause` deriving in-flight from `scheduling_ledger` (D-01 authoritative, never `saq_jobs`), and `saq_detail` reading `saq_jobs` only as a `begin_nested()` SAVEPOINT-isolated corroborating detail that degrades safe; the D-01 written decision record lives in the module docstring (INFLIGHT-03/SC#5). The DERIV-04 anti-drift lock is a parametrized SQL⇔Python equivalence matrix (`tests/integration/test_stage_status_equivalence.py`, 25 cases, real PG green via `just test-bucket integration`) asserting `sql_status == py_status == expected` for every (stage × status) cell; 44 DB-free unit cells in `tests/shared/`. Verifier 12/12 must-haves (independent codebase read). Code-review gate: 0 blockers, 3 warnings / 2 info — WR-01 (`done(review)`==`done(propose)` makes `eligible(review)` unreachable under real derivation) + WR-02 (downstream `eligible()` doesn't gate on `IN_FLIGHT`) are the RESEARCH-flagged deferred assumptions (A3/OQ1, backstopped by deterministic-key dedup); WR-03 (`_DONE_FP` fingerprint allowlist duplicated across the two halves) is a real drift nit the equivalence test would catch at runtime — candidate for `/gsd:code-review 78 --fix` before cutover. Ships as its own PR on a worktree branch. **Next: Phase 79 (Shadow-Compare Gate) — the re-runnable live-corpus implication check that must pass before any reader cutover and before `033`.**
 
 **Phase-by-phase execution history (2026.7.2):**
 
@@ -412,7 +416,9 @@ This document evolves at phase transitions and milestone boundaries.
 4. Update Context with current state
 
 ---
-*Last updated: 2026-07-08 — **milestone 2026.7.5 Parallel Enrich DAG started** (`/gsd:new-milestone`). Retire the linear `FileState` enum entirely; replace it with a derived pure function `stage_status(file, stage) -> {not_started | in_flight | done | failed}` computed from existing sources (output rows / `saq_jobs` / new per-stage failure markers), with eligibility as a pure predicate over it. The three enrich stages become genuinely independent: `eligible = NOT done AND NOT in_flight`, no cross-stage gating. Comprehensive scope — `FileState`'s non-completion jobs move to sidecars (`cloud_job`, a dedup marker, `proposals.status`, `execution_log`); ~20 writers / ~40 readers reworked across 23 source files. Grounded against `main` @ `ce0c6434`: verified that **no file can currently complete all three enrich stages** (analyze pends on `DISCOVERED`, fingerprint on `METADATA_EXTRACTED`, and the metadata callback advances between them). Two-step migration (`032` additive + backfill → shadow-compare gate on the live corpus → `033` destructive) with a live-corpus verification story. Bitmap stays derived (no denormalized column, YAGNI); partial indexes keep the `NOT EXISTS` anti-joins fast at 200K files. Zero new dependencies. Design contract: `milestones/PARALLEL-ENRICH-DAG-DESIGN.md`. Fresh branch `SimplicityGuy/true-parallel`, separate from PR #221. Phases continue from 76 (start at 77). Requirements + roadmap being defined this cycle.*
+*Last updated: 2026-07-08 — **Phase 78 (Derivation Layer, Eligibility & Anti-Drift Test Harness) complete + verified** (12/12 must-haves) in milestone 2026.7.5 Parallel Enrich DAG. Purely additive single-source-of-truth predicate layer: `enums/stage.py` (DB-free, agent-safe — `Stage`/`Status` enums + `ELIGIBILITY_DAG` + `resolve_status()` precedence ladder + `eligible()` predicate, subprocess-enforced import boundary, 100% cov) + `services/stage_status.py` (SQLAlchemy `ColumnElement` twin, `in_flight` from `scheduling_ledger` per D-01, SAVEPOINT-isolated `saq_detail`, D-01 decision record in docstring), locked together by the DERIV-04 SQL⇔Python equivalence matrix (25 real-PG cases + 44 DB-free unit cells). No reader/writer cut over (Phase 79+). Code review 0 blockers / 3 warnings (WR-01/02 = RESEARCH-deferred eligibility assumptions; WR-03 `_DONE_FP` duplication drift-nit). Next: Phase 79 (Shadow-Compare Gate, live corpus).*
+
+*Prior update: 2026-07-08 — **milestone 2026.7.5 Parallel Enrich DAG started** (`/gsd:new-milestone`). Retire the linear `FileState` enum entirely; replace it with a derived pure function `stage_status(file, stage) -> {not_started | in_flight | done | failed}` computed from existing sources (output rows / `saq_jobs` / new per-stage failure markers), with eligibility as a pure predicate over it. The three enrich stages become genuinely independent: `eligible = NOT done AND NOT in_flight`, no cross-stage gating. Comprehensive scope — `FileState`'s non-completion jobs move to sidecars (`cloud_job`, a dedup marker, `proposals.status`, `execution_log`); ~20 writers / ~40 readers reworked across 23 source files. Grounded against `main` @ `ce0c6434`: verified that **no file can currently complete all three enrich stages** (analyze pends on `DISCOVERED`, fingerprint on `METADATA_EXTRACTED`, and the metadata callback advances between them). Two-step migration (`032` additive + backfill → shadow-compare gate on the live corpus → `033` destructive) with a live-corpus verification story. Bitmap stays derived (no denormalized column, YAGNI); partial indexes keep the `NOT EXISTS` anti-joins fast at 200K files. Zero new dependencies. Design contract: `milestones/PARALLEL-ENRICH-DAG-DESIGN.md`. Fresh branch `SimplicityGuy/true-parallel`, separate from PR #221. Phases continue from 76 (start at 77). Requirements + roadmap being defined this cycle.*
 
 *Prior update: 2026-07-06 after the 2026.7.2 milestone — **Multi-Compute Agents SHIPPED + milestone COMPLETED** (phases 72–76, 17 plans, PRs #209/#210/#211/#213/#214). Milestone audit PASSED: 15/15 requirements, 5/5 phases verified & Nyquist-compliant, 6/6 cross-phase integration seams wired, E2E N-compute dispatch→push→reconcile→render flow complete. N cloud-compute agents now dispatch/route/reconcile/fail-isolate simultaneously (compute-side twin of the N-Kueue support), the `≤1-compute` fail-fasts retired, zero new dependencies. Four low-severity/cosmetic review items closed at close-out by quick `260706-odc` (73-WR-03/IN-01 hardening + 74-IN-01/IN-02 cosmetic; 116 tests green). GAP-01 (N-compute-aware orphan recovery) deferred to v2 PROV-01. Archived to `milestones/2026.7.2-{ROADMAP,REQUIREMENTS,MILESTONE-AUDIT}.md`; `REQUIREMENTS.md` removed (fresh for next milestone). Milestone-completion PR #215 open (archival + close-out fixes + `2026.7.1`→`2026.7.2` version bump); the annotated `2026.7.2` tag is applied to the merged `main` commit + pushed (GHCR publish) after #215 lands. No active milestone — next scoped via `/gsd:new-milestone`.*
 

.planning/REQUIREMENTS.md

@@ -15,24 +15,24 @@
 
 ### Derived Status Layer (DERIV)
 
-- [ ] **DERIV-01**: A single predicate module is the one source of truth for every stage's `done` and `failed` predicate, expressed as reusable `ColumnElement[bool]` builders that compose into both SQL `.where(...)` and a Python per-row resolver — no stage predicate is written twice.
-- [ ] **DERIV-02**: A pure function `stage_status(file, stage) -> {not_started | in_flight | done | failed}` returns the derived status for any file/stage, with precedence `in_flight ≻ done ≻ failed ≻ not_started`.
-- [ ] **DERIV-03**: `done` is derived from the stage's output row using the *correct* completion predicate per stage: `metadata` row present (and not failure-only); `fingerprint_results.status IN ('success','completed')` (any engine); `analysis.analysis_completed_at IS NOT NULL` (not bare row existence); `tracklists`/`proposals`/`execution_log` presence for downstream stages.
-- [ ] **DERIV-04**: A parametrized equivalence test proves the SQL-derived status and the Python-derived status agree for every stage across a representative fixture matrix (locks the two definitions against drift).
-- [ ] **DERIV-05**: `stage_status` correctly aggregates multi-row output tables — a file with one `success` and one `failed` fingerprint engine derives `done`, not `failed`.
+- [x] **DERIV-01**: A single predicate module is the one source of truth for every stage's `done` and `failed` predicate, expressed as reusable `ColumnElement[bool]` builders that compose into both SQL `.where(...)` and a Python per-row resolver — no stage predicate is written twice.
+- [x] **DERIV-02**: A pure function `stage_status(file, stage) -> {not_started | in_flight | done | failed}` returns the derived status for any file/stage, with precedence `in_flight ≻ done ≻ failed ≻ not_started`.
+- [x] **DERIV-03**: `done` is derived from the stage's output row using the *correct* completion predicate per stage: `metadata` row present (and not failure-only); `fingerprint_results.status IN ('success','completed')` (any engine); `analysis.analysis_completed_at IS NOT NULL` (not bare row existence); `tracklists`/`proposals`/`execution_log` presence for downstream stages.
+- [x] **DERIV-04**: A parametrized equivalence test proves the SQL-derived status and the Python-derived status agree for every stage across a representative fixture matrix (locks the two definitions against drift).
+- [x] **DERIV-05**: `stage_status` correctly aggregates multi-row output tables — a file with one `success` and one `failed` fingerprint engine derives `done`, not `failed`.
 
 ### Eligibility (ELIG)
 
-- [ ] **ELIG-01**: The three enrich stages (metadata, fingerprint, analyze) are eligible iff `NOT done AND NOT in_flight`, each **independent of every other stage** — every `discovered` file is simultaneously eligible for all three, workable in any order.
-- [ ] **ELIG-02**: Downstream eligibility is a pure predicate over `stage_status`: tracklist = fingerprint-done & not-tracklisted; propose = metadata-done AND analyze-done; review = a proposal exists; apply = an approved proposal exists.
-- [ ] **ELIG-03**: A failed **analyze** is terminal — it is NOT auto-eligible and never re-enqueued by any automatic path (retry is manual-only), with a regression test asserting a failed analyze is absent from the analyze pending set (guards against the 44.5K-job over-enqueue class).
-- [ ] **ELIG-04**: A failed **fingerprint** remains eligible (auto-retry preserved), consistent with today's D-16 behavior.
+- [x] **ELIG-01**: The three enrich stages (metadata, fingerprint, analyze) are eligible iff `NOT done AND NOT in_flight`, each **independent of every other stage** — every `discovered` file is simultaneously eligible for all three, workable in any order.
+- [x] **ELIG-02**: Downstream eligibility is a pure predicate over `stage_status`: tracklist = fingerprint-done & not-tracklisted; propose = metadata-done AND analyze-done; review = a proposal exists; apply = an approved proposal exists.
+- [x] **ELIG-03**: A failed **analyze** is terminal — it is NOT auto-eligible and never re-enqueued by any automatic path (retry is manual-only), with a regression test asserting a failed analyze is absent from the analyze pending set (guards against the 44.5K-job over-enqueue class).
+- [x] **ELIG-04**: A failed **fingerprint** remains eligible (auto-retry preserved), consistent with today's D-16 behavior.
 
 ### In-Flight Detection (INFLIGHT)
 
-- [ ] **INFLIGHT-01**: `in_flight(file, stage)` is true when an active/queued unit of work exists for that `(file, stage-function)`, and it is a first-class input to both eligibility and the DAG busy pills.
-- [ ] **INFLIGHT-02**: Every read of the SAQ `saq_jobs` table is static SQL wrapped in a `begin_nested()` SAVEPOINT and degrades to a safe default on any error — the 5s `/pipeline/stats` poll never 500s; Alembic never references `saq_jobs`.
-- [ ] **INFLIGHT-03**: *(Open decision D-01 — resolve during Phase 78 planning with a written decision record.)* `in_flight`'s authoritative source is chosen between `scheduling_ledger` alone (Architecture's position) and `saq_jobs ∪ scheduling_ledger` (design/Stack position); whichever is chosen, a crashed-mid-run or callback-lost file is not falsely re-enqueued as `not_started`.
+- [x] **INFLIGHT-01**: `in_flight(file, stage)` is true when an active/queued unit of work exists for that `(file, stage-function)`, and it is a first-class input to both eligibility and the DAG busy pills.
+- [x] **INFLIGHT-02**: Every read of the SAQ `saq_jobs` table is static SQL wrapped in a `begin_nested()` SAVEPOINT and degrades to a safe default on any error — the 5s `/pipeline/stats` poll never 500s; Alembic never references `saq_jobs`.
+- [x] **INFLIGHT-03**: *(Open decision D-01 — resolve during Phase 78 planning with a written decision record.)* `in_flight`'s authoritative source is chosen between `scheduling_ledger` alone (Architecture's position) and `saq_jobs ∪ scheduling_ledger` (design/Stack position); whichever is chosen, a crashed-mid-run or callback-lost file is not falsely re-enqueued as `not_started`.
 
 ### Per-Stage Failure (FAIL)
 
@@ -125,18 +125,18 @@
 
 | Requirement | Phase | Status |
 |-------------|-------|--------|
-| DERIV-01 | Phase 78 | Pending |
-| DERIV-02 | Phase 78 | Pending |
-| DERIV-03 | Phase 78 | Pending |
-| DERIV-04 | Phase 78 | Pending |
-| DERIV-05 | Phase 78 | Pending |
-| ELIG-01 | Phase 78 | Pending |
-| ELIG-02 | Phase 78 | Pending |
-| ELIG-03 | Phase 78 | Pending |
-| ELIG-04 | Phase 78 | Pending |
-| INFLIGHT-01 | Phase 78 | Pending |
-| INFLIGHT-02 | Phase 78 | Pending |
-| INFLIGHT-03 | Phase 78 | Pending |
+| DERIV-01 | Phase 78 | Complete |
+| DERIV-02 | Phase 78 | Complete |
+| DERIV-03 | Phase 78 | Complete |
+| DERIV-04 | Phase 78 | Complete |
+| DERIV-05 | Phase 78 | Complete |
+| ELIG-01 | Phase 78 | Complete |
+| ELIG-02 | Phase 78 | Complete |
+| ELIG-03 | Phase 78 | Complete |
+| ELIG-04 | Phase 78 | Complete |
+| INFLIGHT-01 | Phase 78 | Complete |
+| INFLIGHT-02 | Phase 78 | Complete |
+| INFLIGHT-03 | Phase 78 | Complete |
 | FAIL-01 | Phase 81 | Pending |
 | FAIL-02 | Phase 81 | Pending |
 | FAIL-03 | Phase 81 | Pending |