Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
9873c6f
docs(84): capture phase context
SimplicityGuy Jul 10, 2026
b9597fe
docs(state): record phase 84 context session
SimplicityGuy Jul 10, 2026
c115353
docs(84): add research and validation strategy
SimplicityGuy Jul 10, 2026
298ec99
docs(84): add pattern map
SimplicityGuy Jul 10, 2026
3e7a020
docs(84): create phase plan (dedup & fingerprint-progress cutover)
SimplicityGuy Jul 10, 2026
16dae23
docs(84): create phase plan
SimplicityGuy Jul 10, 2026
703ef8a
feat(84-01): add migration 035 reconcile dedup_resolution
SimplicityGuy Jul 10, 2026
69f35e6
test(84-01): add migration 035 reconcile integration test
SimplicityGuy Jul 10, 2026
9e0328b
docs(84-01): complete migration 035 reconcile dedup_resolution plan
SimplicityGuy Jul 10, 2026
0903f12
feat(84-02): add file-level dedup_resolved_clause() to stage_status.py
SimplicityGuy Jul 10, 2026
aac04b2
docs(84-02): record scan_deletion dual-FK un-resolve behavior in mode…
SimplicityGuy Jul 10, 2026
05cc3e5
docs(84-02): complete shared dedup-resolved predicate plan
SimplicityGuy Jul 10, 2026
5f5ab42
feat(84-03): cut dedup.py over to the durable marker (writer + CAS un…
SimplicityGuy Jul 10, 2026
7ed693d
test(84-03): inconsistent-corpus dedup divergence guard across the fi…
SimplicityGuy Jul 10, 2026
a67ed16
test(84-03): resolve→undo→re-resolve shadow-compare integration test …
SimplicityGuy Jul 10, 2026
3da3a65
docs(84-03): complete dedup marker cutover plan
SimplicityGuy Jul 10, 2026
15bd574
feat(84-04): derive get_fingerprint_progress from output tables + ded…
SimplicityGuy Jul 10, 2026
b7ed1a9
test(84-04): replace get_fingerprint_progress mock stub with real-DB …
SimplicityGuy Jul 10, 2026
40bc1bb
docs(84-04): complete get_fingerprint_progress derivation cutover plan
SimplicityGuy Jul 10, 2026
ba63d31
test(84-05): AST source-scan guard for dedup/fingerprint FileState cu…
SimplicityGuy Jul 10, 2026
814d22e
docs(84-05): complete AST source-scan guard plan
SimplicityGuy Jul 10, 2026
3ec4ade
fix(84): revert premature Complete marks on READ-04 / SIDECAR-02
SimplicityGuy Jul 10, 2026
c862d1d
test(84): update router fingerprint-progress test to the derived cont…
SimplicityGuy Jul 10, 2026
463917a
docs(84-06): record live-corpus shadow-compare as a blocking pre-merg…
SimplicityGuy Jul 10, 2026
2a99b79
docs(84): record phase verification (human_needed — 84-06 live run open)
SimplicityGuy Jul 10, 2026
5215d82
fix(84): validate undo payload before deleting the dedup marker (WR-0…
SimplicityGuy Jul 10, 2026
75bb56a
docs(84): record code review findings
SimplicityGuy Jul 10, 2026
eab46d5
docs(84): record resolution of code-review findings WR-01/WR-02
SimplicityGuy Jul 10, 2026
35cf8b2
docs(84): mark code review resolved in report body
SimplicityGuy Jul 10, 2026
62e819e
docs(84-06): close live-corpus gate with a read-only production measu…
SimplicityGuy Jul 10, 2026
19e7118
docs(84-06): remove stale 'not executed' text from superseded deferra…
SimplicityGuy Jul 10, 2026
4f01628
docs(phase-84): complete phase execution
SimplicityGuy Jul 10, 2026
0efbee4
docs(state): correct stale last_activity after phase 84 close
SimplicityGuy Jul 10, 2026
3cf93e0
test(84): close T-84-04-01 and T-84-03-03 with real regression tests
SimplicityGuy Jul 10, 2026
8534b5c
test(84): lock shadow_compare's read-only property (T-84-06-02 harden…
SimplicityGuy Jul 10, 2026
c3c5680
docs(phase-84): add security threat verification (16/16 closed, harde…
SimplicityGuy Jul 10, 2026
1d274ef
test(84): close Nyquist gap — T-84-03-02 undo-payload branch coverage
SimplicityGuy Jul 10, 2026
026b6c0
docs(phase-84): update validation strategy (nyquist_compliant, 1 gap …
SimplicityGuy Jul 10, 2026
48e70f5
test(84): complete UAT - 5 passed, 3 issues
SimplicityGuy Jul 10, 2026
74f1f12
fix(84): commit the dedup resolve/undo transactions (UAT blocker)
SimplicityGuy Jul 10, 2026
7a7dc13
docs(84-06): correct post-deploy prediction — analyzed invariant will…
SimplicityGuy Jul 10, 2026
1988c5f
docs(84): close out UAT — 2 blockers fixed, 1 doc corrected, 1 backlo…
SimplicityGuy Jul 10, 2026
238fc44
docs(84): ship phase 84 — PR #228
SimplicityGuy Jul 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .planning/REQUIREMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,13 @@
- [ ] **READ-01**: The three enrich pending sets are derived from `stage_status` (not from `FileRecord.state`), so metadata/fingerprint/analyze each surface every not-done, not-in-flight file independent of the others — the current cross-stage deadlock is gone (a file can complete all three, in any order).
- [ ] **READ-02**: `get_pipeline_stats` reports per-stage counts from output tables (the linear `GROUP BY FileRecord.state` is removed), and the DAG shows four-bucket per-stage counts (not_started / in_flight / done / failed) including a visible failed count per enrich stage.
- [ ] **READ-03**: Recovery/re-enqueue (`reenqueue.py`, `reconcile_cloud_jobs.py`) derive their done/in-flight sets from `stage_status`/sidecars with no `FileRecord.state` read, preserving the scheduling-ledger recovery contract and the "only previously-scheduled work recovers" guarantee.
- [ ] **READ-04**: Dedup (`services/dedup.py`) and `get_fingerprint_progress` derive from the dedup marker / output tables rather than `FileRecord.state`.
- [x] **READ-04**: Dedup (`services/dedup.py`) and `get_fingerprint_progress` derive from the dedup marker / output tables rather than `FileRecord.state`.
- [ ] **READ-05**: The dead `state == EXECUTED` gates are revived against the real apply-outcome source — tag writing, review, tags/cue/tracklists guards fire for actually-applied files (fixes the permanently-dead tag-writer path).

### Sidecar Migration (SIDECAR)

- [x] **SIDECAR-01**: Cloud-routing status (`AWAITING_CLOUD`/`PUSHING`/`PUSHED`/`LOCAL_ANALYZING`) is represented via the `cloud_job` sidecar (and/or derived `in_flight(analyze)`), with the CAS-guard behavior of `/pushed`, `/mismatch`, and `/upload-failed` preserved or strengthened (closes the missing-CAS-guard bug at `agent_s3.py:195`).
- [ ] **SIDECAR-02**: Dedup resolution (`DUPLICATE_RESOLVED`) is represented via a durable dedup marker, with resolve/undo preserved and backfilled from existing rows.
- [x] **SIDECAR-02**: Dedup resolution (`DUPLICATE_RESOLVED`) is represented via a durable dedup marker, with resolve/undo preserved and backfilled from existing rows.
- [ ] **SIDECAR-03**: Review decisions (approve/reject) and apply outcomes are read from `proposals.status` + `execution_log` — `FileRecord.state` is no longer a redundant, drift-prone mirror of proposal state (fixes the `store_proposals` MOVED-regression bug).

### Operator UI (UI)
Expand Down Expand Up @@ -144,10 +144,10 @@
| READ-01 | Phase 82 | Pending |
| READ-02 | Phase 82 | Pending |
| READ-03 | Phase 80 | Pending |
| READ-04 | Phase 84 | Pending |
| READ-04 | Phase 84 | Complete |
| READ-05 | Phase 85 | Pending |
| SIDECAR-01 | Phase 83 | Complete |
| SIDECAR-02 | Phase 84 | Pending |
| SIDECAR-02 | Phase 84 | Complete |
| SIDECAR-03 | Phase 86 | Pending |
| UI-01 | Phase 87 | Pending |
| UI-02 | Phase 87 | Pending |
Expand Down
26 changes: 22 additions & 4 deletions .planning/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Retire the linear `FileState` enum and derive per-file, per-stage status (`not_s
- [x] **Phase 81: Per-Stage Failure Persistence & Retry Paths** — durable failure markers for analyze + metadata (`report_metadata_failed` records instead of nothing) + reused fingerprint failure; a metadata retry path so a failure is never a permanent dead-end (FAIL-01..04) (completed 2026-07-09)
- [ ] **Phase 82: Counts & Pending-Set Cutover** — the three enrich pending sets + `get_pipeline_stats` derived from `stage_status`; the cross-stage deadlock dissolves; four-bucket per-stage counts; the 200K-scale poll latency measured (READ-01, READ-02, PERF-02)
- [x] **Phase 83: Cloud-Routing Sidecar Cutover** — cloud routing (`AWAITING_CLOUD`/`PUSHING`/`PUSHED`/`LOCAL_ANALYZING`) via the `cloud_job` sidecar / derived `in_flight(analyze)`, one atomic consistency domain, CAS-guard collapse (closes the missing `/upload-failed` guard) (SIDECAR-01) (completed 2026-07-09)
- [ ] **Phase 84: Dedup & Fingerprint-Progress Cutover** — `services/dedup.py` + `get_fingerprint_progress` derive from the dedup marker / output tables; resolve/undo preserved (READ-04, SIDECAR-02)
- [x] **Phase 84: Dedup & Fingerprint-Progress Cutover** — `services/dedup.py` + `get_fingerprint_progress` derive from the dedup marker / output tables; resolve/undo preserved (READ-04, SIDECAR-02) (completed 2026-07-10)
- [ ] **Phase 85: EXECUTED-Gate Revival** — the dead `state == EXECUTED` gates revived against the real apply-outcome (`applied(f)` predicate); turns tag/CUE writing on for the first time — **own PR, live-UAT-worthy, not bundled** (READ-05)
- [ ] **Phase 86: Proposals Cutover** — `proposals.status` becomes the sole authority; the redundant `FileRecord.state` cascade (`_TERMINAL_FILE_STATES`) deleted, dissolving the `store_proposals` MOVED-regression bug (SIDECAR-03)
- [ ] **Phase 87: Operator UI — Stage Matrix, Failure Retry, Eligibility Trace & Priority** — per-file derived stage matrix (paginated), per-stage failure visibility + retry, the "why not eligible?" trace, force-done/skip, orphaned-work count, and the restored per-stage priority stepper (UI-01..05, PRIO-01)
Expand Down Expand Up @@ -272,7 +272,7 @@ Deployment-gated verification deferred to the live OCI A1 rollout (see STATE.md
| 81. Per-Stage Failure Persistence & Retry Paths | 2026.7.5 | 6/6 | Complete | 2026-07-09 |
| 82. Counts & Pending-Set Cutover | 2026.7.5 | 0/0 | Not started | - |
| 83. Cloud-Routing Sidecar Cutover | 2026.7.5 | 7/7 | Complete | 2026-07-09 |
| 84. Dedup & Fingerprint-Progress Cutover | 2026.7.5 | 0/0 | Not started | - |
| 84. Dedup & Fingerprint-Progress Cutover | 2026.7.5 | 6/6 | Complete | 2026-07-10 |
| 85. EXECUTED-Gate Revival | 2026.7.5 | 0/0 | Not started | - |
| 86. Proposals Cutover | 2026.7.5 | 0/0 | Not started | - |
| 87. Operator UI — Stage Matrix, Failure Retry, Eligibility Trace & Priority | 2026.7.5 | 0/0 | Not started | - |
Expand Down Expand Up @@ -436,15 +436,33 @@ Plans:
### Phase 84: Dedup & Fingerprint-Progress Cutover

**Goal**: Cut `services/dedup.py` and `get_fingerprint_progress` over to the dedup marker / output tables, so dedup resolve/undo and the fingerprint progress bar derive from data rather than `FileRecord.state`.
**Depends on**: Phase 82
**Depends on**: Phase 77, Phase 78, Phase 79 <!-- corrected 2026-07-09: was "Phase 82" (stale). Verified during plan-phase: file sets are disjoint (82 owns services/pipeline.py; 84 owns services/dedup.py + services/fingerprint.py), no phase-82 branch exists, and 84's real upstreams are 77 (dedup_resolution table + ix_fprint_success), 78 (stage_status predicate module), 79 (shadow gate) — all merged on main. -->
**Requirements**: READ-04, SIDECAR-02
**Success Criteria** (what must be TRUE):

1. Dedup's exclusion filters and resolve/undo read/write the durable dedup marker (undo becomes a plain `DELETE`), with resolve/undo preserved and backfilled rows honored — no `FileRecord.state` read.
2. `get_fingerprint_progress` derives from the per-engine coverage predicate / output tables, not `FileRecord.state`.
3. The shadow-compare gate stays green after the cutover.

**Plans**: TBD
**Plans**: 6 plans in 4 waves
Plans:
**Wave 1**

- [x] 84-01-PLAN.md — Migration `035` reconcile (bidirectional dedup marker vs files.state) + migration test (SIDECAR-02) [wave 1]
- [x] 84-02-PLAN.md — D-13 `dedup_resolved_clause()` predicate in stage_status.py + D-08 model docstring note (READ-04, SIDECAR-02) [wave 1]

**Wave 2** *(depends on 84-01, 84-02)*

- [x] 84-03-PLAN.md — dedup.py cutover: pg_insert writer + DELETE…RETURNING CAS undo + nine reader flips + divergence & shadow-compare tests (READ-04, SIDECAR-02) [wave 2]
- [x] 84-04-PLAN.md — get_fingerprint_progress rewrite over MUSIC_VIDEO_TYPES + marker + done/failed_clause + real-DB replacement test (READ-04) [wave 2]

**Wave 3** *(depends on 84-03, 84-04)*

- [x] 84-05-PLAN.md — AST source-scan guard over dedup.py + fingerprint.py, mutation-tested both directions (READ-04, SIDECAR-02) [wave 3]

**Wave 4** *(depends on 84-03, 84-04, 84-05 — operator checkpoint)*

- [x] 84-06-PLAN.md — Live-corpus `just shadow-compare` run after `035`, assert hard_fail_total=0 (SIDECAR-02) [wave 4]

### Phase 85: EXECUTED-Gate Revival

Expand Down
29 changes: 14 additions & 15 deletions .planning/STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,15 @@
gsd_state_version: 1.0
milestone: 2026.7.5
milestone_name: Parallel Enrich DAG
status: ready_to_plan
last_updated: "2026-07-10T00:03:11.000Z"
last_activity: 2026-07-09 -- Phase 83 shipped (PR #227)
status: "Phase 84 shipped — PR #228"
last_updated: "2026-07-10T05:29:03.367Z"
last_activity: 2026-07-09
progress:
total_phases: 52
completed_phases: 19
total_plans: 76
completed_plans: 20
percent: 37
stopped_at: Phase 83 shipped as PR #227 (verified 7/7) — ready to discuss Phase 84
completed_phases: 20
total_plans: 82
completed_plans: 26
percent: 38
---

# Project State
Expand All @@ -21,20 +20,20 @@ stopped_at: Phase 83 shipped as PR #227 (verified 7/7) — ready to discuss Phas
See: .planning/PROJECT.md (updated 2026-07-06 — 2026.7.2 Multi-Compute Agents shipped)

**Core value:** Get 200K messy music and concert files properly named, organized, deduplicated, with rich metadata in Postgres -- human-in-the-loop approval so nothing moves without review. Files stay on file-server agents; decisions stay on the application server.
**Current focus:** Phase 84dedup & fingerprint progress cutover
**Current focus:** Phase 85executed gate revival

## Current Position

Phase: 84
Phase: 85
Plan: Not started
Status: Ready to plan (Phase 83 shipped — PR #227, awaiting CI + merge)
Status: Phase 84 shipped — PR #228
Last activity: 2026-07-09

## Performance Metrics

**v1.0 Velocity:**

- Total plans completed: 194
- Total plans completed: 200
- Total phases: 11
- Timeline: 4 days (2026-03-27 -> 2026-03-30)
- Tests: 282 passing
Expand Down Expand Up @@ -290,9 +289,9 @@ These are tracked follow-ups; none blocks the 2026.7.1 milestone record. The PRO

## Session Continuity

Last session: 2026-07-09T18:50:41.889Z
Stopped at: Phase 83 context gathered
Resume file: .planning/phases/83-cloud-routing-sidecar-cutover/83-CONTEXT.md
Last session: 2026-07-10T00:37:11.136Z
Stopped at: Phase 84 context gathered
Resume file: .planning/phases/84-dedup-fingerprint-progress-cutover/84-CONTEXT.md

## Operator Next Steps

Expand Down
Loading
Loading