Symptom
On 2026-07-02, a merged change (#2728, haku-console) took ~2.5h to actually roll out despite passing CI in minutes. Every devel push during that window — including the ducktape-automation[bot] "sync artifact pins" commits — cancelled the in-flight CI run before its push-images job could publish. ~12 consecutive runs were cancelled at the push-images stage; the console image only published once the merge train paused long enough for one full run to complete.
Root cause
.github/workflows/ci.yml:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress is correct for bazel-ci — no point burning RBE on a commit that's already superseded. But the same group also covers push-images (needs: [bazel-ci]), so a new push kills image publishing for the previous commit too. On a quiet day the next run republishes; on a busy day the train never pauses and nothing publishes. The bot's own artifact-pin syncs make it self-perpetuating.
Options (operator decision — tradeoffs)
- Separate concurrency group for
push-images (e.g. group: push-images-${{ github.ref }}, cancel-in-progress: false) so publishing a superseded commit's images finishes instead of being killed. Cost: transiently publishes images for intermediate commits (Flux image automation already picks the newest tag, so this is mostly wasted work, not incorrectness).
- Merge queue — serialize devel merges so there's no train to begin with. Bigger change; also fixes the RBE waste.
- Accept as-is — rollouts are eventually-consistent and this only bites during merge storms.
No urgency; filing so the choice is deliberate rather than rediscovered. Observed while shipping #2728.
Filed by Haku (background agent) per operator request.
Symptom
On 2026-07-02, a merged change (#2728, haku-console) took ~2.5h to actually roll out despite passing CI in minutes. Every
develpush during that window — including theducktape-automation[bot]"sync artifact pins" commits — cancelled the in-flight CI run before itspush-imagesjob could publish. ~12 consecutive runs were cancelled at thepush-imagesstage; the console image only published once the merge train paused long enough for one full run to complete.Root cause
.github/workflows/ci.yml:cancel-in-progressis correct forbazel-ci— no point burning RBE on a commit that's already superseded. But the same group also coverspush-images(needs: [bazel-ci]), so a new push kills image publishing for the previous commit too. On a quiet day the next run republishes; on a busy day the train never pauses and nothing publishes. The bot's own artifact-pin syncs make it self-perpetuating.Options (operator decision — tradeoffs)
push-images(e.g.group: push-images-${{ github.ref }},cancel-in-progress: false) so publishing a superseded commit's images finishes instead of being killed. Cost: transiently publishes images for intermediate commits (Flux image automation already picks the newest tag, so this is mostly wasted work, not incorrectness).No urgency; filing so the choice is deliberate rather than rediscovered. Observed while shipping #2728.
Filed by Haku (background agent) per operator request.