Skip to content

Phase 83: Cloud-Routing Sidecar Cutover#227

Merged
SimplicityGuy merged 54 commits into
mainfrom
SimplicityGuy/phaze-83
Jul 10, 2026
Merged

Phase 83: Cloud-Routing Sidecar Cutover#227
SimplicityGuy merged 54 commits into
mainfrom
SimplicityGuy/phaze-83

Conversation

@SimplicityGuy

Copy link
Copy Markdown
Owner

Summary

Phase 83: Cloud-Routing Sidecar Cutover
Goal: Cloud routing (AWAITING_CLOUD/PUSHING/PUSHED/LOCAL_ANALYZING) via the cloud_job sidecar / derived in_flight(analyze), one atomic consistency domain, CAS-guard collapse (closes the missing /upload-failed guard).
Status: Verified ✓ (7/7 must-haves)

Moves cloud routing off the linear FileRecord.state column and onto the cloud_job sidecar table as the single source of truth. Every AWAITING_CLOUD file now carries a cloud_job(status='awaiting') row — written by one shared writer, backfilled for the existing corpus, and reaped when analyze reaches a terminal state. All four push/upload callback CAS guards collapse onto cloud_job.status as the single CAS domain, which closes the long-standing missing-guard bug at agent_s3.py:195. The drain reader is cut over behind a hard two-tick double-dispatch gate.

FileRecord.state is still dual-written (D-00c) — only reliance on it is retired here. The column itself dies in Phase 90.

Changes

83-01: Shared awaiting-cloud writer

hold_awaiting_cloud() — the single go-forward writer of cloud_job.status='awaiting'. Never commits (the caller owns the commit boundary); 'awaiting' is deliberately outside IN_FLIGHT so a re-stamped row never inflates a backend's in-flight count.

Key files: src/phaze/services/backends.py

83-02: Migration 034 — existing-corpus backfill

Data-only repair backfilling the missing sidecar rows for every file parked in AWAITING_CLOUD since migration 032. Static, parameter-free SQL; idempotent via ON CONFLICT DO NOTHING; empty autogenerate diff; read-only on files.state.

Key files: alembic/versions/034_backfill_cloud_awaiting.py

83-03: D-14 awaiting-row reaper

Deletes the inert cloud_job row at both analyze-terminal seams (put_analysis, report_analysis_failed), bounding ix_cloud_job_awaiting growth. Without it, a locally-dispatched long file keeps an awaiting row forever and the */5 drain tick scans a permanently growing dead set at 200K files.

Key files: src/phaze/routers/agent_analysis.py

83-04: Push/upload CAS cutover

All four callback guards re-anchored on cloud_job.status. report_upload_failed gains a CAS guard, a FULL no-op on CAS miss, and a pg_advisory_xact_lock on its attempt RMW. report_pushed/report_push_mismatch swap their anchor off FileRecord.state == PUSHING onto cloud_job.status == 'submitted'.

Behavior change: /mismatch now requires a submitted compute cloud_job to spill — the old "no-cloud_job file spills on state == PUSHING" path is intentionally gone (compute-only, D-12).

Key files: src/phaze/routers/agent_s3.py, src/phaze/routers/agent_push.py

83-05: Hold-path cutover

trigger_analysis swaps its bare file.state = AWAITING_CLOUD for hold_awaiting_cloud(). The go-forward half of the invariant; migration 034 is the existing-corpus half. Phase-79 shadow gate proven green on a held-file fixture.

Key files: src/phaze/routers/pipeline.py

83-06: Drain reader cutover

get_cloud_staging_candidates excludes in-flight/terminally-failed local analyze by a predicate conjunct (not row deletion), locks candidacy with with_for_update(of=CloudJob, skip_locked=True), keeps FIFO on created_at while moving the staleness clock to cloud_job.updated_at. get_awaiting_cloud_count re-anchored on the same clause.

Key files: src/phaze/services/pipeline.py, src/phaze/services/backend_selection.py, src/phaze/tasks/release_awaiting_cloud.py

83-07: Single-writer consolidation (gap closure)

Closes the one must-have the first verification failed. hold_awaiting_cloud becomes a dual-mode writer: unconditional upsert for the hold path, rowcount-guarded CAS for the spills. Both over-cap spill paths now route through it, so no inline awaiting re-stamp survives in either router.

Key files: src/phaze/services/backends.py, src/phaze/routers/agent_s3.py, src/phaze/routers/agent_push.py, tests/analyze/services/test_single_awaiting_writer.py

Requirements Addressed

  • SIDECAR-01 — Cloud-routing status (AWAITING_CLOUD/PUSHING/PUSHED/LOCAL_ANALYZING) is represented via the cloud_job sidecar (and/or derived in_flight(analyze)), with the CAS-guard behavior of /pushed, /mismatch, and /upload-failed preserved or strengthened (closes the missing-CAS-guard bug at agent_s3.py:195).

Verification

  • Automated verification: passed, 7/7 must-haves (83-VERIFICATION.md)
  • SC#1 — no FileRecord.state routing read remains in the drain path or any of the four callback CAS guards. Display/stats reads (get_pipeline_stats) are unaffected and in scope by design.
  • SC#2report_upload_failed CAS guard + advisory lock, backed by a discriminating regression test that fails if the guard is removed.
  • SC#3 (HARD GATE)tests/integration/test_drain_double_dispatch.py, 3/3 cases. Committed RED (cases b/c genuinely fail against the state-based drain), turned GREEN by the cutover. Not skipped, not xfail'd.
  • Full suite: 2142 passed, 0 failed across agents/integration/shared/analyze.
  • ruff check clean · ruff format clean · mypy clean (205 source files).
  • Migration 034 applies cleanly on a fresh DB; alembic current = 034 (head); downgrade documented-lossy.
  • Phase-79 shadow-compare gate stays green; services/shadow_compare.py untouched.

No human verification items. The live-corpus shadow-compare run and EXPLAIN (ANALYZE, BUFFERS) query-plan check remain explicitly deferred, non-blocking items per 83-VALIDATION.md.

Key Decisions

  • D-05 — conjunct over deletion. The drain excludes in-flight/terminally-failed analyze by predicate, not by deleting the sidecar row. Row deletion would reintroduce a rolled-back-tick double-dispatch; SC#3's case (b) exercises exactly that hazard.
  • D-03 — spills re-stamp to awaiting, not FAILED. Over-cap spills keep attempts spent (cloud_submit_max_attempts) as the budget marker select_backend reads to route the file to local. 'awaiting' ∉ IN_FLIGHT.
  • D-07 — staleness clock moved to the sidecar. FIFO stays on FileRecord.created_at; staleness reads cloud_job.updated_at, so it survives Phase 90's removal of files.state.
  • D-11 / Landmine L1 — transaction discipline. hold_awaiting_cloud never commits: a commit would drop the tick's pg_advisory_xact_lock and reopen the over-stage class.
  • D-02 — one writer, enforced by a test, not a docstring. The gap-closure plan ships an AST guard asserting hold_awaiting_cloud is the sole writer of status='awaiting'. It was mutation-tested against three reintroduction forms (literal keyword, **splat, subscript mutation) and goes RED on each, while not false-positiving on the drain's .where() readers, the D-14 reaper's delete().where(), or an unrelated .values(**row) elsewhere.

Reviewer Notes

Two things worth a reviewer's attention, both known and tracked:

  1. hold_awaiting_cloud leaves cloud_phase NULL on the hold path, so get_analyze_stage_files derives lane "a1" (compute) instead of "local" for held and locally-spilled files. This is a display-only regression introduced by this phase — no routing or data-integrity impact — and is deliberately scoped out of the gap-closure plan. No test covers that lane derivation, which is why it slipped through. Tracked as WR-01 in 83-REVIEW.md.

  2. trigger_backfill_cloud seeds a process_file:<id> recovery-ledger row for compute-target held files, which the new ~inflight_clause(ANALYZE) drain conjunct now excludes. A backfill-held compute file therefore falls to local analysis rather than the compute backend — mis-routed, not stranded. No test exercises backfill → drain, so the suite stays green. Logged in deferred-items.md under ## 83-06 for the backfill/recovery owner.

Also carried forward: a pre-existing non-hermetic test-setup flake (pk_agents duplicate-key under random ordering) in the analyze and agents buckets. Reproduces with this phase's tests deselected; not introduced here. Logged for a hygiene task.

🤖 Generated with Claude Code

SimplicityGuy and others added 30 commits July 9, 2026 11:50
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
… waves)

Wave 1 (parallel): 83-01 shared hold_awaiting_cloud() writer · 83-02
migration 034 corpus repair · 83-03 D-14 awaiting-row reaper.
Wave 2 (parallel): 83-04 callback CAS collapse (SC#2) · 83-05 hold-path
cutover + shadow green.
Wave 3: 83-06 drain reader cutover + count card + staleness clock + SC#3
HARD GATE.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
- Single go-forward writer of cloud_job.status='awaiting' (D-01/D-02)
- Dual-writes FileState.AWAITING_CLOUD (D-00c) + upserts sidecar on file_id
- on_conflict_do_update re-stamps a terminalized row, attempts from arg (D-03)
- Never commits (dispatch discipline; preserves tick advisory lock, L1)
…oud_job rows

- Re-runs 032's _BACKFILL_CLOUD_AWAITING verbatim (INSERT..SELECT 'awaiting' ON CONFLICT DO NOTHING)
- Sync, static parameter-free SQL, chains down_revision=033
- Touches no ORM schema (awaiting CHECK + ix_cloud_job_awaiting shipped in 032) so autogenerate stays empty
- Documented-lossy downgrade DELETEs awaiting rows
…al seams

- put_analysis (success) and report_analysis_failed (terminal) now DELETE the
  file's cloud_job row WHERE status='awaiting', joining each seam's existing txn
- bounds ix_cloud_job_awaiting growth (D-14) so the */5 drain tick never scans a
  monotonically growing dead set; SUCCEEDED/RUNNING rows left untouched
- import CloudJobStatus; no new session/commit; AUTH-01 preserved (path file_id)
…gration 034

- Seeds row-less awaiting_cloud files, a non-awaiting control, and a pre-existing cloud_job row
- Asserts one awaiting row per held file, zero for control, unchanged pre-existing row (DO NOTHING)
- Re-runs backfill to prove idempotency (uq_cloud_job_file_id, no duplicate rows)
- Asserts empty autogenerate diff (034 is data-only) and files.state byte-unchanged
- Static-SQL parameter-free assertion (T-83-03); documented-lossy downgrade removes awaiting rows
…touched)

- put_analysis success reaps an awaiting cloud_job row; report_analysis_failed
  terminal reaps an awaiting cloud_job row
- SUCCEEDED and RUNNING cloud_job rows are left untouched by the terminal reaper
  (status='awaiting' filter), proving cloud-analyzed rows survive
- add _seed_cloud_job / _cloud_job_present helpers (staging_bucket NULL so the
  seam's S3 guard short-circuits); reuse the existing seed_test_agent scaffolds
…on, D-13 flip

- Test A: fresh hold writes one awaiting row + flips state, no commit needed
- Test B: spill re-stamp of a FAILED row stays one row, retains spent attempts (D-03)
- Test C: 'awaiting' not in backends.IN_FLIGHT (D-03 no in-flight inflation)
- Test D: LocalBackend.dispatch keeps the inert awaiting row, flips LOCAL_ANALYZING (D-13)
- Note the pre-existing order-dependent analyze-bucket flake in deferred-items.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
# Conflicts:
#	.planning/phases/83-cloud-routing-sidecar-cutover/deferred-items.md
…-01)

- Replace bare file.state = FileState.AWAITING_CLOUD with await hold_awaiting_cloud(session, file)
- Import hold_awaiting_cloud from phaze.services.backends
- Every go-forward long-file hold now carries its cloud_job(status='awaiting', attempts=0) sidecar row
- Post-loop commit retained (the helper never commits)
…ting row (D-00d)

- Assert AWAITING_CLOUD file + cloud_job(status='awaiting') yields zero HARD awaiting_cloud divergence
  (the shape the go-forward writer + migration 034 guarantee; violated at HEAD without them)
- Assert a LOCAL_ANALYZING file carrying an inert awaiting row (post-D-13, pre-D-14 reap) drives no
  hard-invariant flag -- implication-not-equality (79 D-04), no converse invariant
- No change to services/shadow_compare.py
… (RED)

Five new failing regressions + one updated existing assertion, all RED against
current source:
- test_upload_failed_cas_noop_on_advanced_cloud_job (SC#2/T-83-01): over-cap
  /upload-failed on a running/succeeded cloud_job is a FULL no-op.
- test_failed_concurrent_under_cap_no_lost_update (D-11/T-83-02): two concurrent
  /upload-failed increment s3_upload_attempt to exactly 2 (RED: lost update -> 1).
- test_pushed_does_not_clobber_when_cloud_job_not_submitted (SC#1): late /pushed
  on an already-advanced cloud_job cannot clobber.
- test_push_mismatch_over_cap_spill_restamps_cloud_job_to_awaiting (D-03): spill
  re-stamps 'submitted' -> 'awaiting' (RED: current writes 'failed').
- test_push_mismatch_over_cap_does_not_clobber_when_cloud_job_not_submitted (SC#1):
  late over-cap /mismatch on an already-advanced cloud_job cannot spill.
- updated test_upload_failed_at_cap_spills... to assert AWAITING (D-03), not FAILED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
…y lock (D-09/D-10/D-11/D-03)

- D-11: pg_advisory_xact_lock(hashtext(ledger_key)) before the s3_upload_attempt
  RMW read serializes concurrent /failed (no lost update, no self-deadlock vs the
  before_enqueue hook's session) -- copied verbatim from agent_push.py:240.
- D-09: the over-cap spill CAS anchors on cloud_job.status IN ('uploading','uploaded')
  (the sidecar, not FileRecord.state) -- closes the unguarded :195 clobber (SC#2/T-83-01).
- D-03: the CAS re-stamps to status='awaiting' (was FAILED) so AWAITING_CLOUD =>
  status='awaiting' holds, keeping attempts spent + cloud_phase=None.
- D-10: rowcount==0 is a FULL no-op (no FileRecord write, no abort/delete_staged_object,
  no ledger clear); the FileRecord dual-write + S3 cleanup + ledger clear are gated
  behind rowcount!=0.

SC#2 no-op + D-11 concurrency regressions go GREEN; full agent_s3 test file green.
…s=='submitted' (SC#1/D-12/D-03)

- The over-cap spill CAS now anchors on cloud_job.status == 'submitted' (compute's
  single in-flight status, D-12) instead of FileRecord.state == PUSHING -- removing
  a FileRecord.state routing read (SC#1) before Phase 90 drops the column.
- D-03: the SAME CAS re-stamps submitted -> awaiting (was a separate FAILED write),
  so AWAITING_CLOUD => status='awaiting' holds; the redundant FAILED UPDATE is deleted.
- rowcount==0 -> FULL no-op; the FileRecord dual-write (a plain write, not a
  state==PUSHING predicate) + ledger clear are gated behind rowcount!=0.
- The :240 advisory lock and the D-07 reporter-auth 403 are unchanged, still before
  the CAS; the under-cap re-drive path is byte-unchanged.

D-03 awaiting spill + /mismatch clobber regressions go GREEN; existing over-cap tests
updated to seed a 'submitted' cloud_job and assert the awaiting re-stamp.
…bmitted' (SC#1/D-12)

- report_pushed's CAS now anchors on cloud_job.status == 'submitted' (compute's
  single in-flight status, D-12) instead of FileRecord.state == PUSHING -- removing
  the last cloud-routing FileRecord.state routing read (SC#1) before Phase 90.
- The single CAS replaces BOTH the old FileRecord PUSHING->PUSHED guard AND the
  unconditional cloud_job SUCCEEDED write: it terminalizes SUBMITTED->SUCCEEDED and
  gates the FileRecord PUSHED dual-write + ledger clear + process_file enqueue behind
  its rowcount.
- Safe on the 'submitted' literal despite kueue also transiting SUBMITTED: a kueue
  file returns via the :107 no-attributed-backend 200 hold before the CAS. No
  backend-kind check and no reporter==agent_ref gate added (D-12); :107 hold unchanged.

Late/duplicate /pushed clobber regression goes GREEN; full agents bucket green in
isolation (450 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
SimplicityGuy and others added 24 commits July 9, 2026 13:26
- Drives two sequential stage_cloud_window ticks across (a) local dispatch,
  (b) rolled-back tick with a committed ledger row, (c) terminally-failed analyze
- Asserts each file dispatched exactly once, never cloud after a local dispatch
- RED against the current FileRecord.state drain: cases (b) and (c) re-pick /
  cloud-dispatch a file the sidecar drain (D-05/D-06/D-07) must exclude; case (a)
  passes under both (happy path)
…-07, GREEN)

- get_cloud_staging_candidates: INNER join cloud_job WHERE status='awaiting'
  AND ~inflight_clause(ANALYZE) AND ~domain_completed_clause(ANALYZE), FOR UPDATE
  OF cloud_job SKIP LOCKED; FIFO stays on FileRecord.created_at; surfaces
  cloud_job.updated_at as the lane-entry staleness clock. No FileRecord.state read (SC#1)
- select_backend: staleness clock reads the passed lane_entered_at (cloud_job.updated_at)
  instead of file.updated_at (D-07, Phase-90-durable)
- SC#3 HARD GATE now GREEN across all three cases (exactly-once, never cloud after local)
- Update existing drain/dispatch tests to seed the cloud_job(status='awaiting') sidecar row
  + ledger rows where before_enqueue would write them
…D-15)

- get_awaiting_cloud_count now COUNT(cloud_job) WHERE status='awaiting'
  AND ~inflight_clause(ANALYZE) AND ~domain_completed_clause(ANALYZE), INNER-joined to
  FileRecord so the correlated clause builders resolve; kept _safe_count-wrapped. Card and
  drain derive from the SAME clause -> cannot disagree. No FileRecord.state read (SC#1)
- get_pushing_count / get_pushed_count untouched (unowned Phase-90 blocker)
- Add D-15 unit test (locally-analyzing awaiting file excluded; parked file counted; count
  == drain candidate set) + update existing count-card tests to seed the awaiting sidecar row
… writer

- Add expect_status / clear_cloud_phase kwargs; return bool
- Hold mode (expect_status=None): unconditional upsert + D-00c file.state dual-write, returns True
- Spill mode: rowcount-guarded CAS only, no FileRecord write, never commits (D-09/D-10/D-11/D-12)
- clear_cloud_phase gates the cloud_phase=None write (s3 spill only, D-12)
- Unit tests: hold-branch return, spill CAS hit/miss no-op, cloud_phase preserve-vs-clear
- report_upload_failed (agent_s3): inline CAS -> hold_awaiting_cloud(expect_status=(uploading,uploaded), clear_cloud_phase=True)
- report_push_mismatch (agent_push): inline CAS -> hold_awaiting_cloud(expect_status=(submitted,)) (no cloud_phase touch, D-12)
- NULL-GUARD: absent FileRecord takes the FULL no-op (cleared=False), never AttributeError; helper not called
- grep -c CloudJobStatus.AWAITING.value now 0 in both routers; behavior byte-identical (39 pre-existing tests green)
- Add NULL-GUARD test to each router module
…waiting writer

- Hermetic (no-DB) ast.parse scan of src/phaze/**.py
- Flags any .values(status=AWAITING...) WRITE outside services/backends.py
- Keys on .values(...) so drain/count/shadow WHERE-reads + D-14 reaper DELETE are NOT flagged
- Goes RED if a router re-introduces an inline awaiting re-stamp (D-02 self-enforcing)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
The D-02 anti-drift scan only matched the literal `.values(status=...)`
keyword form. `ast.keyword.arg` is None for a `**splat`, so an inline
awaiting writer using `.values(**vals)` -- the idiom the allowlisted
writer itself uses in backends.py -- passed the guard GREEN. Confirmed
by mutation test: injecting a splat-form inline writer into
routers/agent_push.py did not turn the test RED.

Now resolves `**name` splats to their dict-literal binding (and to
`vals["status"] = ...` subscript mutations), and flags unresolvable
splats only when the statement targets CloudJob and the module
references AWAITING -- so services/proposal.py's unrelated
`.values(**row)` does not false-positive.

Verified by mutation: literal-keyword, dict-splat, and subscript forms
each turn the guard RED; clean source stays GREEN.

Found by 83-07 code review (WR-01).
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@SimplicityGuy SimplicityGuy merged commit 6855cfe into main Jul 10, 2026
31 checks passed
@SimplicityGuy SimplicityGuy deleted the SimplicityGuy/phaze-83 branch July 10, 2026 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant