Phase 81: Per-Stage Failure Persistence & Retry Paths

[SimplicityGuy]

Summary

Phase 81: Per-Stage Failure Persistence & Retry Paths
Goal: Make all three enrich stages persist a durable failure marker and gain a retry path — closing the latent bug where a failed metadata extraction records nothing and becomes invisible-and-permanently-ineligible.
Status: Verified ✓ (4/4 must-haves)

Before this phase, report_metadata_failed persisted nothing. A terminally-failed metadata extraction left no row, so derivation could not see it, counts could not report it, and no operator surface could retry it — the file was invisible and permanently ineligible, blocking it from ever reaching propose. Phase 81 closes that (gap G-01, CRITICAL) and brings the three enrich stages onto one coherent failure model.

analyze and metadata now stamp a durable failed_at + error_message marker via pg_insert(...).on_conflict_do_update, and clear it unconditionally on a later success. analyze's writer dual-writes the marker alongside the legacy files.state = ANALYSIS_FAILED write in one transaction, so recovery can derive from the marker while the three live state readers keep working until their cutover. fingerprint is deliberately left alone — it already persists fingerprint_results.status='failed' per engine, and this phase adds only characterization tests plus docstrings locking that asymmetry in (FAIL-04 asks for reused, not re-invented). Migration 033 makes the analyze invariant structural with a analysis_completed_at XOR failed_at CHECK, preceded in the same upgrade() by the set-based cleanup UPDATE that makes the live corpus satisfy it. Finally, POST /pipeline/metadata-failed/retry gives the operator a bulk retry path that re-enqueues every failed-metadata file and, per D-11, leaves the failure row in place until a real success clears it.

The two terminality axes are now named tables in the DB-free enums/stage.py — FAILURE_IS_TERMINAL and ELIGIBLE_AFTER_FAILURE — so no second reader ever re-derives the analyze/fingerprint asymmetry by hand. domain_completed() ships with its SQL twin domain_completed_clause() in the same phase, drift-locked by a parametrized equivalence test.

Changes

Plan 81-01: DB-free terminality tables + domain_completed twin

Two orthogonal per-stage tables (recovery terminality vs auto-retry eligibility), a pure domain_completed() predicate, its ColumnElement SQL twin, and a semantics-preserving eligible() refactor.

Key files: src/phaze/enums/stage.py, src/phaze/services/stage_status.py, tests/integration/test_stage_status_equivalence.py

Plan 81-02: Migration 033 — mixed-row cleanup then XOR CHECK

ck_analysis_analysis_completed_xor_failed enforced at the DB and mirrored into AnalysisResult.__table_args__. The D-09 cleanup UPDATE runs before create_check_constraint so Phase 77's unguarded backfill rows don't abort the migration; done wins (failed_at cleared, analysis_completed_at retained). The destructive Phase-90 migration is renumbered 033 → 034 in forward-looking docs.

Key files: alembic/versions/033_add_analysis_completed_xor_failed.py, src/phaze/models/analysis.py, tests/integration/test_migrations/test_migration_033_additive_check.py

Plan 81-03: Metadata failure persistence (FAIL-02, the latent bug)

report_metadata_failed upserts a durable failure row instead of recording nothing. The body is optional (MetadataFailurePayload | None = None), so an old bodyless agent image still gets a 200 and still clears its ledger row (D-10 version-skew guard). put_metadata clears the marker unconditionally on success, including the empty-body branch (D-13).

Key files: src/phaze/routers/agent_metadata.py, src/phaze/schemas/agent_metadata.py, src/phaze/services/agent_client.py, src/phaze/tasks/metadata_extraction.py

Plan 81-04: Fingerprint asymmetry, locked by characterization tests

No new writer. Regression tests assert report_fingerprint_failed persists no row, never writes a synthetic engine='_task' sentinel (which would poison the per-engine joins), and that a status='failed' engine row keeps the file eligible(fingerprint).

Key files: src/phaze/routers/agent_fingerprint.py, tests/fingerprint/routers/test_agent_fingerprint_failure.py

Plan 81-05: Analyze failure dual-write

report_analysis_failed stamps the marker and the legacy state write in one transaction. It uses ON CONFLICT DO UPDATE rather than a bare UPDATE, because a pure analyze failure never wrote an analysis row at all — an UPDATE would silently no-op and lose the marker. put_analysis clears failed_at/error_message outside exclude_unset, so a real success both restores the D-06 invariant and satisfies the new CHECK.

Key files: src/phaze/routers/agent_analysis.py, tests/analyze/routers/test_agent_analysis_failure.py

Plan 81-06: Metadata operator retry path (FAIL-03)

POST /pipeline/metadata-failed/retry mirrors retry_analysis_failed's Phase-30 guard ordering (resolve the per-agent queue once → NoActiveAgentError guard → enqueue with the complete ExtractMetadataPayload), minus the state flip — metadata has no terminal FileState. Backed by get_metadata_failed_files, a correlated exists() on metadata.failed_at IS NOT NULL.

Key files: src/phaze/routers/pipeline.py, src/phaze/services/pipeline.py, src/phaze/templates/pipeline/partials/metadata_retry_response.html, tests/integration/routers/test_pipeline_metadata_retry.py

Requirements Addressed

Req	Description	Plans
FAIL-01	analyze failures persist a durable failure marker with an error reason, backfilled from existing ANALYSIS_FAILED rows	81-01, 81-02, 81-05
FAIL-02	metadata failures persist a durable failure marker — report_metadata_failed records the failure instead of nothing (closes the latent bug where terminally-failed metadata is invisible everywhere)	81-03
FAIL-03	A terminally-failed metadata file has a retry path, so a metadata failure is never a permanent dead-end that blocks the file from ever reaching propose (closes gap G-01, CRITICAL)	81-06
FAIL-04	fingerprint failure continues to persist via fingerprint_results.status='failed' (reused, not re-invented) and stays auto-retryable	81-01, 81-04

4 requirements, 0 orphans.

Verification

• Automated verification: PASSED — 4/4 must-haves verified (81-VERIFICATION.md), each traced to source and to independently re-run tests rather than cited from SUMMARY.
• Code review (81-REVIEW.md) — 20 files, 2 criticals found and fixed before close: CR-01 retry_analysis_failed did not clear analysis.failed_at (1ff92265); CR-02 the domain_completed twins rejected downstream stages asymmetrically (a6398d33).
• Security audit (81-SECURITY.md) — 24/24 threats closed, register authored at plan time. Two remediated during the audit: T-81-03-04/T-81-05-03 (1d6af9f7), where a NUL in an agent's error text aborted the transaction that also clears the scheduling ledger, stranding the file in an unbounded recovery loop. The audit also surfaced WR-03 (feaebc48): eligible() compared Status by identity, so a raw-string status map — exactly what a SQL round-trip yields — reported a terminally-failed analyze as eligible, the same class as the 44.5K-job over-enqueue incident.
• Nyquist validation (81-VALIDATION.md) — 16 task rows audited, all green in bucket isolation. One gap found and filled: 81-SECURITY.md closed T-81-03-04/T-81-05-03 on two limbs, but the max_length=2000 limb had no test in either payload — deleting the bound would have left the suite green while re-opening both threats (365a467b).

Test evidence (every bucket run in isolation, not only as part of the full suite):

Bucket	Result
shared	997 passed
metadata	77 passed
fingerprint	83 passed
analyze	519 passed
integration	155 passed
docs-drift	10 passed

uv run mypy clean across all six touched source modules.

Key Decisions

• D-04 / D-09 — done wins. Migration 033's cleanup UPDATE runs before create_check_constraint, clearing failed_at and retaining analysis_completed_at on mixed rows. Matches Phase 78's _analyze_status done-over-failed precedence. A validating DDL constraint is preceded, in the same upgrade(), by the UPDATE that makes the existing corpus satisfy it.
• D-06 — the invariant is structural. No analysis row may ever carry both analysis_completed_at and failed_at. Enforced by CHECK, mirrored in the ORM, and satisfied by both writers.
• D-10 — version-skew safety. The failure endpoint's body is optional. A control plane at Phase 81 with an agent still on the prior image must return 200 and still clear the ledger, or the file strands.
• D-11 — retry does not clear the marker. retry_metadata_failed re-enqueues without touching the failure row; only put_metadata's clear-on-success wipes it. A retry that hasn't succeeded yet must still read as failed.
• D-18 — fingerprint is reused, not re-invented. report_fingerprint_failed persists no row, by design; FAILURE_IS_TERMINAL[fingerprint] = False makes auto-retry intentional rather than accidental.
• Two orthogonal axes, named once. FAILURE_IS_TERMINAL (does recovery consider this stage finished?) and ELIGIBLE_AFTER_FAILURE (may it auto-retry?) are separate tables. Collapsing them is what let the analyze/fingerprint asymmetry get re-derived inconsistently by different readers.
• Python predicate + SQL twin ship together. domain_completed() and domain_completed_clause() land in the same phase, drift-locked by a parametrized equivalence test — closing the window Phase 78's D-04 left open.

Known Open Items (deliberate, tracked)

Two review warnings are carried forward in deferred-items.md rather than fixed here. Both are real, both are scoped to Phase 80's reader cutover, and neither blocks this phase's goal:

• WR-01 — report_metadata_failed's conflict branch does not null pre-existing payload columns, so a file with real tags that later fails extraction derives FAILED while still holding usable metadata.
• WR-02 — domain_completed_clause has no inflight disjunct while the Python twin ranks IN_FLIGHT above FAILED, so the twins diverge on in_flight ∧ failed rows — a cell that FAIL-03's new retry path makes reachable. The equivalence test's DOMAIN_COMPLETED_CASES deliberately excludes the in-flight seeds, so it will not catch this. Recorded under Known Coverage Boundaries in 81-VALIDATION.md so a green suite is not mistaken for a stronger guarantee than it makes.

Two manual verifications remain (both need infrastructure a test DB can't reproduce): the bodyless-POST path against a genuinely older deployed agent image, and migration 033's cleanup against a restored production snapshot.

Note on Diff Scope

This branch also carries c3cf84b7, a planning-only ROADMAP edit rewiring Phase 80/83 depends_on to break a 80 → 83 → 82 → 80 dependency cycle. It touches no source. The .planning/ tree accounts for 26 of the 51 changed files; source + migration + template changes total 15 files (+521/−70).

🤖 Generated with Claude Code

https://claude.ai/code/session_01LM2oA27zEky9dq3PVjJteL

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!