Skip to content

CI: cancel-in-progress starves push-images on busy merge days — image rollouts stall for hours #2759

Description

@agentydragon

Symptom

On 2026-07-02, a merged change (#2728, haku-console) took ~2.5h to actually roll out despite passing CI in minutes. Every devel push during that window — including the ducktape-automation[bot] "sync artifact pins" commits — cancelled the in-flight CI run before its push-images job could publish. ~12 consecutive runs were cancelled at the push-images stage; the console image only published once the merge train paused long enough for one full run to complete.

Root cause

.github/workflows/ci.yml:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

cancel-in-progress is correct for bazel-ci — no point burning RBE on a commit that's already superseded. But the same group also covers push-images (needs: [bazel-ci]), so a new push kills image publishing for the previous commit too. On a quiet day the next run republishes; on a busy day the train never pauses and nothing publishes. The bot's own artifact-pin syncs make it self-perpetuating.

Options (operator decision — tradeoffs)

  1. Separate concurrency group for push-images (e.g. group: push-images-${{ github.ref }}, cancel-in-progress: false) so publishing a superseded commit's images finishes instead of being killed. Cost: transiently publishes images for intermediate commits (Flux image automation already picks the newest tag, so this is mostly wasted work, not incorrectness).
  2. Merge queue — serialize devel merges so there's no train to begin with. Bigger change; also fixes the RBE waste.
  3. Accept as-is — rollouts are eventually-consistent and this only bites during merge storms.

No urgency; filing so the choice is deliberate rather than rediscovered. Observed while shipping #2728.

Filed by Haku (background agent) per operator request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions