refactor(controller): canary reconcile pipeline with guaranteed status sync. Fixes #4626

[zachaller]

Summary

• Phase 1: Guarantee syncRolloutStatusCanary() runs after every canary actuation pass so evaluateProgressDeadlineAbort, progress deadline timeout, and requeueStuckRollout cannot be skipped when a stage errors (progressDeadlineSeconds/progressDeadlineAbort silently non-functional on canary rollouts with Istio traffic routing when canary ReplicaSet is unavailable #4626). Ports fix(controller)!: do not block abort/progressDeadline on traffic routing errors. Fixes #4626 #4962's weight-verification safety (stable scale-down hold, shouldFullPromote verified-weight check, end-of-rollout unverified-weight handling).
• Phase 2: Replace per-pass deferral flags with persisted rollout conditions (TrafficRoutingApplied, ServicesReconciled, ActuationSucceeded) merged during status sync; progression gates read current-pass stage condition state.
• Phase 3: Replace the hand-rolled canary stage chain with a declarative stage table + executor. Traffic routing errors use stageHold (status sync + verify-interval requeue, no reconcile error); other failures remain fatal with workqueue backoff. Centralizes condition recording and TrafficRoutingError event emission.

Supersedes #4962 — same #4626 fix, generalized to all canary actuation stages.

Behavior change (release note)

Rollouts stuck on a persistently failing traffic router (or other actuation stage) will now evaluate progressDeadlineAbort every reconcile and can time out / auto-abort when progressDeadlineAbort: true. Previously they could stay Progressing forever.

Test plan

• go test ./rollout/... ./utils/conditions/...
• make test-e2e — TestCanarySuite, TestFunctionalSuite
• make test-e2e — Istio progress-deadline abort scenario (optional; covered in fix(controller)!: do not block abort/progressDeadline on traffic routing errors. Fixes #4626 #4962)

Made with Cursor

[zachaller]

Follow up work:

1. Blue-green: apply to rolloutBlueGreen (bluegreen.go:19). Extra design needed: syncRolloutStatusBlueGreen(previewSvc, activeSvc) requires the services, and the services-fetch is the first stage — define a degraded status-sync path when it fails (e.g. sync conditions/deadline only, skip service-derived fields).
2. Downgrade cosmetic stages: make reconcileEphemeralMetadata and reconcileRevisionHistoryLimit stageHold-like (continue the whole pipeline, not just status sync) — nothing downstream depends on them for traffic safety. One-line table change; needs its own release-note.
3. Service-switch persistent failures: consider stageHold (not Fatal) with a ServicesReconciled gate — the strongest remaining progressDeadlineSeconds/progressDeadlineAbort silently non-functional on canary rollouts with Istio traffic routing when canary ReplicaSet is unavailable #4626-class production failure mode (admission webhook / RBAC on Services). Requires the same "skip downstream traffic routing" reasoning as fix(controller)!: do not block abort/progressDeadline on traffic routing errors. Fixes #4626 #4962; the pipeline makes this a table + skip-rule change.
4. Unify step plugin hasError into the condition mechanism fully.
5. Revisit the pod-restart stageStopNoStatus anomaly: should a restart-heavy pass still evaluate progressDeadline? Probably yes, but it changes deadline behavior for restart storms — needs its own analysis.

[github-actions]

Published E2E Test Results

  4 files    4 suites   4h 7m 28s ⏱️
150 tests 136 ✅  7 💤 7 ❌
608 runs  572 ✅ 28 💤 8 ❌

For more details on these failures, see this check.

Results for commit ea191c9.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 643 tests   2 643 ✅  3m 30s ⏱️
  131 suites      0 💤
    1 files        0 ❌

Results for commit ea191c9.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 84.21053% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.22%. Comparing base (f2c5c2b) to head (ea191c9).

Files with missing lines	Patch %	Lines
rollout/stages.go	78.57%	16 Missing and 8 partials ⚠️
rollout/stageconditions.go	92.50%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4963      +/-   ##
==========================================
+ Coverage   85.17%   85.22%   +0.04%     
==========================================
  Files         166      168       +2     
  Lines       19453    19569     +116     
==========================================
+ Hits        16570    16677     +107     
- Misses       2030     2038       +8     
- Partials      853      854       +1     

Flag	Coverage Δ
e2e	53.21% <77.77%> (+0.18%)	⬆️
unit-tests	81.66% <73.09%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud