Phase 83: Cloud-Routing Sidecar Cutover#227
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
… waves) Wave 1 (parallel): 83-01 shared hold_awaiting_cloud() writer · 83-02 migration 034 corpus repair · 83-03 D-14 awaiting-row reaper. Wave 2 (parallel): 83-04 callback CAS collapse (SC#2) · 83-05 hold-path cutover + shadow green. Wave 3: 83-06 drain reader cutover + count card + staleness clock + SC#3 HARD GATE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
- Single go-forward writer of cloud_job.status='awaiting' (D-01/D-02) - Dual-writes FileState.AWAITING_CLOUD (D-00c) + upserts sidecar on file_id - on_conflict_do_update re-stamps a terminalized row, attempts from arg (D-03) - Never commits (dispatch discipline; preserves tick advisory lock, L1)
…oud_job rows - Re-runs 032's _BACKFILL_CLOUD_AWAITING verbatim (INSERT..SELECT 'awaiting' ON CONFLICT DO NOTHING) - Sync, static parameter-free SQL, chains down_revision=033 - Touches no ORM schema (awaiting CHECK + ix_cloud_job_awaiting shipped in 032) so autogenerate stays empty - Documented-lossy downgrade DELETEs awaiting rows
…al seams - put_analysis (success) and report_analysis_failed (terminal) now DELETE the file's cloud_job row WHERE status='awaiting', joining each seam's existing txn - bounds ix_cloud_job_awaiting growth (D-14) so the */5 drain tick never scans a monotonically growing dead set; SUCCEEDED/RUNNING rows left untouched - import CloudJobStatus; no new session/commit; AUTH-01 preserved (path file_id)
…gration 034 - Seeds row-less awaiting_cloud files, a non-awaiting control, and a pre-existing cloud_job row - Asserts one awaiting row per held file, zero for control, unchanged pre-existing row (DO NOTHING) - Re-runs backfill to prove idempotency (uq_cloud_job_file_id, no duplicate rows) - Asserts empty autogenerate diff (034 is data-only) and files.state byte-unchanged - Static-SQL parameter-free assertion (T-83-03); documented-lossy downgrade removes awaiting rows
…touched) - put_analysis success reaps an awaiting cloud_job row; report_analysis_failed terminal reaps an awaiting cloud_job row - SUCCEEDED and RUNNING cloud_job rows are left untouched by the terminal reaper (status='awaiting' filter), proving cloud-analyzed rows survive - add _seed_cloud_job / _cloud_job_present helpers (staging_bucket NULL so the seam's S3 guard short-circuits); reuse the existing seed_test_agent scaffolds
…on, D-13 flip - Test A: fresh hold writes one awaiting row + flips state, no commit needed - Test B: spill re-stamp of a FAILED row stays one row, retains spent attempts (D-03) - Test C: 'awaiting' not in backends.IN_FLIGHT (D-03 no in-flight inflation) - Test D: LocalBackend.dispatch keeps the inert awaiting row, flips LOCAL_ANALYZING (D-13) - Note the pre-existing order-dependent analyze-bucket flake in deferred-items.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
# Conflicts: # .planning/phases/83-cloud-routing-sidecar-cutover/deferred-items.md
…-01) - Replace bare file.state = FileState.AWAITING_CLOUD with await hold_awaiting_cloud(session, file) - Import hold_awaiting_cloud from phaze.services.backends - Every go-forward long-file hold now carries its cloud_job(status='awaiting', attempts=0) sidecar row - Post-loop commit retained (the helper never commits)
…ting row (D-00d) - Assert AWAITING_CLOUD file + cloud_job(status='awaiting') yields zero HARD awaiting_cloud divergence (the shape the go-forward writer + migration 034 guarantee; violated at HEAD without them) - Assert a LOCAL_ANALYZING file carrying an inert awaiting row (post-D-13, pre-D-14 reap) drives no hard-invariant flag -- implication-not-equality (79 D-04), no converse invariant - No change to services/shadow_compare.py
… (RED) Five new failing regressions + one updated existing assertion, all RED against current source: - test_upload_failed_cas_noop_on_advanced_cloud_job (SC#2/T-83-01): over-cap /upload-failed on a running/succeeded cloud_job is a FULL no-op. - test_failed_concurrent_under_cap_no_lost_update (D-11/T-83-02): two concurrent /upload-failed increment s3_upload_attempt to exactly 2 (RED: lost update -> 1). - test_pushed_does_not_clobber_when_cloud_job_not_submitted (SC#1): late /pushed on an already-advanced cloud_job cannot clobber. - test_push_mismatch_over_cap_spill_restamps_cloud_job_to_awaiting (D-03): spill re-stamps 'submitted' -> 'awaiting' (RED: current writes 'failed'). - test_push_mismatch_over_cap_does_not_clobber_when_cloud_job_not_submitted (SC#1): late over-cap /mismatch on an already-advanced cloud_job cannot spill. - updated test_upload_failed_at_cap_spills... to assert AWAITING (D-03), not FAILED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
…y lock (D-09/D-10/D-11/D-03)
- D-11: pg_advisory_xact_lock(hashtext(ledger_key)) before the s3_upload_attempt
RMW read serializes concurrent /failed (no lost update, no self-deadlock vs the
before_enqueue hook's session) -- copied verbatim from agent_push.py:240.
- D-09: the over-cap spill CAS anchors on cloud_job.status IN ('uploading','uploaded')
(the sidecar, not FileRecord.state) -- closes the unguarded :195 clobber (SC#2/T-83-01).
- D-03: the CAS re-stamps to status='awaiting' (was FAILED) so AWAITING_CLOUD =>
status='awaiting' holds, keeping attempts spent + cloud_phase=None.
- D-10: rowcount==0 is a FULL no-op (no FileRecord write, no abort/delete_staged_object,
no ledger clear); the FileRecord dual-write + S3 cleanup + ledger clear are gated
behind rowcount!=0.
SC#2 no-op + D-11 concurrency regressions go GREEN; full agent_s3 test file green.
…s=='submitted' (SC#1/D-12/D-03) - The over-cap spill CAS now anchors on cloud_job.status == 'submitted' (compute's single in-flight status, D-12) instead of FileRecord.state == PUSHING -- removing a FileRecord.state routing read (SC#1) before Phase 90 drops the column. - D-03: the SAME CAS re-stamps submitted -> awaiting (was a separate FAILED write), so AWAITING_CLOUD => status='awaiting' holds; the redundant FAILED UPDATE is deleted. - rowcount==0 -> FULL no-op; the FileRecord dual-write (a plain write, not a state==PUSHING predicate) + ledger clear are gated behind rowcount!=0. - The :240 advisory lock and the D-07 reporter-auth 403 are unchanged, still before the CAS; the under-cap re-drive path is byte-unchanged. D-03 awaiting spill + /mismatch clobber regressions go GREEN; existing over-cap tests updated to seed a 'submitted' cloud_job and assert the awaiting re-stamp.
…bmitted' (SC#1/D-12) - report_pushed's CAS now anchors on cloud_job.status == 'submitted' (compute's single in-flight status, D-12) instead of FileRecord.state == PUSHING -- removing the last cloud-routing FileRecord.state routing read (SC#1) before Phase 90. - The single CAS replaces BOTH the old FileRecord PUSHING->PUSHED guard AND the unconditional cloud_job SUCCEEDED write: it terminalizes SUBMITTED->SUCCEEDED and gates the FileRecord PUSHED dual-write + ledger clear + process_file enqueue behind its rowcount. - Safe on the 'submitted' literal despite kueue also transiting SUBMITTED: a kueue file returns via the :107 no-attributed-backend 200 hold before the CAS. No backend-kind check and no reporter==agent_ref gate added (D-12); :107 hold unchanged. Late/duplicate /pushed clobber regression goes GREEN; full agents bucket green in isolation (450 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
- Drives two sequential stage_cloud_window ticks across (a) local dispatch, (b) rolled-back tick with a committed ledger row, (c) terminally-failed analyze - Asserts each file dispatched exactly once, never cloud after a local dispatch - RED against the current FileRecord.state drain: cases (b) and (c) re-pick / cloud-dispatch a file the sidecar drain (D-05/D-06/D-07) must exclude; case (a) passes under both (happy path)
…-07, GREEN) - get_cloud_staging_candidates: INNER join cloud_job WHERE status='awaiting' AND ~inflight_clause(ANALYZE) AND ~domain_completed_clause(ANALYZE), FOR UPDATE OF cloud_job SKIP LOCKED; FIFO stays on FileRecord.created_at; surfaces cloud_job.updated_at as the lane-entry staleness clock. No FileRecord.state read (SC#1) - select_backend: staleness clock reads the passed lane_entered_at (cloud_job.updated_at) instead of file.updated_at (D-07, Phase-90-durable) - SC#3 HARD GATE now GREEN across all three cases (exactly-once, never cloud after local) - Update existing drain/dispatch tests to seed the cloud_job(status='awaiting') sidecar row + ledger rows where before_enqueue would write them
…D-15) - get_awaiting_cloud_count now COUNT(cloud_job) WHERE status='awaiting' AND ~inflight_clause(ANALYZE) AND ~domain_completed_clause(ANALYZE), INNER-joined to FileRecord so the correlated clause builders resolve; kept _safe_count-wrapped. Card and drain derive from the SAME clause -> cannot disagree. No FileRecord.state read (SC#1) - get_pushing_count / get_pushed_count untouched (unowned Phase-90 blocker) - Add D-15 unit test (locally-analyzing awaiting file excluded; parked file counted; count == drain candidate set) + update existing count-card tests to seed the awaiting sidecar row
… writer - Add expect_status / clear_cloud_phase kwargs; return bool - Hold mode (expect_status=None): unconditional upsert + D-00c file.state dual-write, returns True - Spill mode: rowcount-guarded CAS only, no FileRecord write, never commits (D-09/D-10/D-11/D-12) - clear_cloud_phase gates the cloud_phase=None write (s3 spill only, D-12) - Unit tests: hold-branch return, spill CAS hit/miss no-op, cloud_phase preserve-vs-clear
- report_upload_failed (agent_s3): inline CAS -> hold_awaiting_cloud(expect_status=(uploading,uploaded), clear_cloud_phase=True) - report_push_mismatch (agent_push): inline CAS -> hold_awaiting_cloud(expect_status=(submitted,)) (no cloud_phase touch, D-12) - NULL-GUARD: absent FileRecord takes the FULL no-op (cleared=False), never AttributeError; helper not called - grep -c CloudJobStatus.AWAITING.value now 0 in both routers; behavior byte-identical (39 pre-existing tests green) - Add NULL-GUARD test to each router module
…waiting writer - Hermetic (no-DB) ast.parse scan of src/phaze/**.py - Flags any .values(status=AWAITING...) WRITE outside services/backends.py - Keys on .values(...) so drain/count/shadow WHERE-reads + D-14 reaper DELETE are NOT flagged - Goes RED if a router re-introduces an inline awaiting re-stamp (D-02 self-enforcing)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013KQB1pKA9PwCMdau7VKPuZ
The D-02 anti-drift scan only matched the literal `.values(status=...)` keyword form. `ast.keyword.arg` is None for a `**splat`, so an inline awaiting writer using `.values(**vals)` -- the idiom the allowlisted writer itself uses in backends.py -- passed the guard GREEN. Confirmed by mutation test: injecting a splat-form inline writer into routers/agent_push.py did not turn the test RED. Now resolves `**name` splats to their dict-literal binding (and to `vals["status"] = ...` subscript mutations), and flags unresolvable splats only when the statement targets CloudJob and the module references AWAITING -- so services/proposal.py's unrelated `.values(**row)` does not false-positive. Verified by mutation: literal-keyword, dict-splat, and subscript forms each turn the guard RED; clean source stays GREEN. Found by 83-07 code review (WR-01).
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 83: Cloud-Routing Sidecar Cutover
Goal: Cloud routing (
AWAITING_CLOUD/PUSHING/PUSHED/LOCAL_ANALYZING) via thecloud_jobsidecar / derivedin_flight(analyze), one atomic consistency domain, CAS-guard collapse (closes the missing/upload-failedguard).Status: Verified ✓ (7/7 must-haves)
Moves cloud routing off the linear
FileRecord.statecolumn and onto thecloud_jobsidecar table as the single source of truth. EveryAWAITING_CLOUDfile now carries acloud_job(status='awaiting')row — written by one shared writer, backfilled for the existing corpus, and reaped when analyze reaches a terminal state. All four push/upload callback CAS guards collapse ontocloud_job.statusas the single CAS domain, which closes the long-standing missing-guard bug atagent_s3.py:195. The drain reader is cut over behind a hard two-tick double-dispatch gate.FileRecord.stateis still dual-written (D-00c) — only reliance on it is retired here. The column itself dies in Phase 90.Changes
83-01: Shared awaiting-cloud writer
hold_awaiting_cloud()— the single go-forward writer ofcloud_job.status='awaiting'. Never commits (the caller owns the commit boundary);'awaiting'is deliberately outsideIN_FLIGHTso a re-stamped row never inflates a backend's in-flight count.Key files:
src/phaze/services/backends.py83-02: Migration 034 — existing-corpus backfill
Data-only repair backfilling the missing sidecar rows for every file parked in
AWAITING_CLOUDsince migration 032. Static, parameter-free SQL; idempotent viaON CONFLICT DO NOTHING; empty autogenerate diff; read-only onfiles.state.Key files:
alembic/versions/034_backfill_cloud_awaiting.py83-03: D-14 awaiting-row reaper
Deletes the inert
cloud_jobrow at both analyze-terminal seams (put_analysis,report_analysis_failed), boundingix_cloud_job_awaitinggrowth. Without it, a locally-dispatched long file keeps anawaitingrow forever and the*/5drain tick scans a permanently growing dead set at 200K files.Key files:
src/phaze/routers/agent_analysis.py83-04: Push/upload CAS cutover
All four callback guards re-anchored on
cloud_job.status.report_upload_failedgains a CAS guard, a FULL no-op on CAS miss, and apg_advisory_xact_lockon its attempt RMW.report_pushed/report_push_mismatchswap their anchor offFileRecord.state == PUSHINGontocloud_job.status == 'submitted'.Behavior change:
/mismatchnow requires asubmittedcomputecloud_jobto spill — the old "no-cloud_job file spills onstate == PUSHING" path is intentionally gone (compute-only, D-12).Key files:
src/phaze/routers/agent_s3.py,src/phaze/routers/agent_push.py83-05: Hold-path cutover
trigger_analysisswaps its barefile.state = AWAITING_CLOUDforhold_awaiting_cloud(). The go-forward half of the invariant; migration 034 is the existing-corpus half. Phase-79 shadow gate proven green on a held-file fixture.Key files:
src/phaze/routers/pipeline.py83-06: Drain reader cutover
get_cloud_staging_candidatesexcludes in-flight/terminally-failed local analyze by a predicate conjunct (not row deletion), locks candidacy withwith_for_update(of=CloudJob, skip_locked=True), keeps FIFO oncreated_atwhile moving the staleness clock tocloud_job.updated_at.get_awaiting_cloud_countre-anchored on the same clause.Key files:
src/phaze/services/pipeline.py,src/phaze/services/backend_selection.py,src/phaze/tasks/release_awaiting_cloud.py83-07: Single-writer consolidation (gap closure)
Closes the one must-have the first verification failed.
hold_awaiting_cloudbecomes a dual-mode writer: unconditional upsert for the hold path, rowcount-guarded CAS for the spills. Both over-cap spill paths now route through it, so no inline awaiting re-stamp survives in either router.Key files:
src/phaze/services/backends.py,src/phaze/routers/agent_s3.py,src/phaze/routers/agent_push.py,tests/analyze/services/test_single_awaiting_writer.pyRequirements Addressed
AWAITING_CLOUD/PUSHING/PUSHED/LOCAL_ANALYZING) is represented via thecloud_jobsidecar (and/or derivedin_flight(analyze)), with the CAS-guard behavior of/pushed,/mismatch, and/upload-failedpreserved or strengthened (closes the missing-CAS-guard bug atagent_s3.py:195).Verification
83-VERIFICATION.md)FileRecord.staterouting read remains in the drain path or any of the four callback CAS guards. Display/stats reads (get_pipeline_stats) are unaffected and in scope by design.report_upload_failedCAS guard + advisory lock, backed by a discriminating regression test that fails if the guard is removed.tests/integration/test_drain_double_dispatch.py, 3/3 cases. Committed RED (cases b/c genuinely fail against the state-based drain), turned GREEN by the cutover. Not skipped, not xfail'd.agents/integration/shared/analyze.ruff checkclean ·ruff formatclean ·mypyclean (205 source files).alembic current=034 (head); downgrade documented-lossy.services/shadow_compare.pyuntouched.No human verification items. The live-corpus shadow-compare run and
EXPLAIN (ANALYZE, BUFFERS)query-plan check remain explicitly deferred, non-blocking items per83-VALIDATION.md.Key Decisions
awaiting, notFAILED. Over-cap spills keepattemptsspent (cloud_submit_max_attempts) as the budget markerselect_backendreads to route the file to local.'awaiting' ∉ IN_FLIGHT.FileRecord.created_at; staleness readscloud_job.updated_at, so it survives Phase 90's removal offiles.state.hold_awaiting_cloudnever commits: a commit would drop the tick'spg_advisory_xact_lockand reopen the over-stage class.hold_awaiting_cloudis the sole writer ofstatus='awaiting'. It was mutation-tested against three reintroduction forms (literal keyword,**splat, subscript mutation) and goes RED on each, while not false-positiving on the drain's.where()readers, the D-14 reaper'sdelete().where(), or an unrelated.values(**row)elsewhere.Reviewer Notes
Two things worth a reviewer's attention, both known and tracked:
hold_awaiting_cloudleavescloud_phaseNULL on the hold path, soget_analyze_stage_filesderives lane"a1"(compute) instead of"local"for held and locally-spilled files. This is a display-only regression introduced by this phase — no routing or data-integrity impact — and is deliberately scoped out of the gap-closure plan. No test covers that lane derivation, which is why it slipped through. Tracked as WR-01 in83-REVIEW.md.trigger_backfill_cloudseeds aprocess_file:<id>recovery-ledger row for compute-target held files, which the new~inflight_clause(ANALYZE)drain conjunct now excludes. A backfill-held compute file therefore falls to local analysis rather than the compute backend — mis-routed, not stranded. No test exercises backfill → drain, so the suite stays green. Logged indeferred-items.mdunder## 83-06for the backfill/recovery owner.Also carried forward: a pre-existing non-hermetic test-setup flake (
pk_agentsduplicate-key under random ordering) in theanalyzeandagentsbuckets. Reproduces with this phase's tests deselected; not introduced here. Logged for a hygiene task.🤖 Generated with Claude Code