CNTRLPLANE-3631: Predictable NodePool Rollout Control#2042
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@csrwng: This pull request references CNTRLPLANE-3631 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
| 1. An automated image-updater bumps the HAProxy image digest in the HyperShift | ||
| operator deployment. | ||
| 2. The operator reconciles all NodePools. The full `Hash()` changes (because it | ||
| includes HAProxy), so a new user-data secret is generated with the latest |
There was a problem hiding this comment.
why is a new userdata secret generated here if there's no consumers for it?
Besides, as soon as a new userdata is created, the existing one gets deleted via token.cleanupOutdated
which would make the nodepool unable to perform scaling operations.
And the token secret gets an expiration timestamp (IgnitionServerTokenExpirationTimestampAnnotation), which will result in the payload being deleted from the ignition server
There was a problem hiding this comment.
You're right — this is the same issue I addressed in my response to your comment on the workflow section header. The current wording in step 2 is wrong: when only management-side content changes and no rollout is triggered, the controller should NOT create new token/user-data secrets or run cleanupOutdated(). The existing secrets remain valid and the MachineDeployment continues to reference them.
I'll rewrite this workflow to reflect the corrected behavior:
- Operator reconciles all NodePools. The
RolloutHashWithoutVersion()does NOT change. - No new token or user-data secrets are created. The existing secrets remain valid.
- The
ConfigUpdatePendingcondition transitions toTruewith reasonManagementConfigDrift. - No MachineDeployment or MachineSet spec change occurs. Existing nodes remain undisturbed. Scale-up continues to work using the existing user-data secret.
| `nodePoolCurrentRolloutConfig` annotation. | ||
| 4. The MachineDeployment spec is updated with the new version and | ||
| `DataSecretName`, triggering a CAPI rolling Replace. | ||
| 5. When the rollout completes (`MachineDeploymentComplete()`), the controller |
There was a problem hiding this comment.
can we articulate the steps for what happens if during a rollout the service provider config changes again.
There was a problem hiding this comment.
Good question. I'll add a fourth workflow scenario covering this. The behavior:
Mid-rollout management-side config change:
- A spec-driven rollout is in progress — the MachineDeployment is rolling,
UpdatingConfigcondition isTrue. - While the rollout is running, a management-side change occurs (e.g., HAProxy image bump).
- The rollout hash has NOT changed (management-side content is excluded), so no new rollout is triggered.
- The in-progress rollout continues using the existing token/user-data secrets. No new secrets are created (rollout hash unchanged).
- When the rollout completes, the
nodePoolCurrentRolloutConfigannotation is updated to the current rollout hash. The nodes that were just replaced have the ignition payload that was generated at the start of the rollout — they do NOT automatically pick up the mid-rollout management-side change. - The
ConfigUpdatePendingcondition may transition toTrueif the management-side change means the current payload differs from what newly-created nodes would get on a fresh provision.
Mid-rollout spec-driven config change:
This is an existing behavior and is unchanged by this enhancement. CAPI handles this via the MachineDeployment's rolling update strategy — the new desired state supersedes the in-progress one, and CAPI continues rolling until all machines match the latest template.
|
|
||
| ### Goals | ||
|
|
||
| 1. Management-plane image digest bumps (e.g., HAProxy, CPO) MUST NOT trigger |
There was a problem hiding this comment.
can we be specific about what things owned by the platform might cause a rollout today? haproxy dataplane image/overrides. What else?
It's also probably worth mentioning that haproxy dataplane image bumping is a particular exception because of early haproxy needs. There's no reason for that image to not come from payload now, unless we ever support shared ingress in selfhosted.
Also should probably include as still a risk to be addressed separately that some config.openshift.io changes to defaults might result in accidental intent for a rollout
There was a problem hiding this comment.
Good points. I'll update the Goals section to enumerate the specific platform-owned inputs that can trigger rollouts today:
-
HAProxy data plane image — the kube-apiserver-proxy static pod image. For shared ingress clusters (ROSA HCP, ARO HCP), this comes from the operator's
IMAGE_SHARED_INGRESS_HAPROXYenv var rather than the NodePool's release payload. For non-shared-ingress (self-hosted) clusters, it already comes from the NodePool's release payload (haproxy-routercomponent), so it's not an issue there. This is a historical artifact from early shared ingress bootstrapping — there's no reason it can't come from the payload now. -
Registry overrides applied to the HAProxy image —
--registry-overrideson the management cluster rewrites the image reference that gets embedded in the ignition payload, even though data plane CRI-O handles mirroring natively (tracked in OCPBUGS-86415). -
config.openshift.iocomputed defaults — theglobalConfigString()function reconciles proxy and image configs with platform-specific defaults (e.g.,Status.NoProxyentries like network CIDRs,169.254.169.254for AWS). If the operator code changes these defaults, the serialized config changes and triggers a rollout even though the user's spec didn't change.
For (3), the two-hash architecture can address this by hashing only the user's raw spec inputs (HostedCluster.Spec.Configuration.Proxy, HostedCluster.Spec.Configuration.Image) in the rollout hash — without reconciliation or computed defaults. The full reconciled config continues to be used for payload generation. This follows the same pattern as HAProxy: user intent drives rollouts, platform-computed values flow into the payload silently.
I'll update the enhancement to cover this as part of the rollout hash design rather than calling it out as a separate risk.
| not yet applied to existing nodes. | ||
|
|
||
| ### Workflow Description | ||
|
|
There was a problem hiding this comment.
can we articulate how any workflow impact the lifecycle of the token secrets and payload cache generation and expiration
There was a problem hiding this comment.
Good call — this is a gap in the current design that needs to be addressed. Let me trace the problem:
When only management-side content changes (e.g., HAProxy bump):
Hash()changes →isOutdated()returns truecleanupOutdated()expires the old token secret (2hr TTL) and deletes the old user-data secret- New token/user-data secrets are created with names based on the new
Hash() - No rollout triggered (rollout hash unchanged) → MachineDeployment still references the old (now deleted) user-data secret
- Scale-up would reference a non-existent secret
The fix: when the rollout hash has not changed, the controller should not create new token/user-data secrets or cleanup existing ones. The existing secrets remain valid — they contain a working ignition payload, and the MachineDeployment continues to reference them. Scale-up nodes get the same config as existing nodes, which is the correct behavior since we're explicitly choosing not to roll out.
For spec-driven rollouts, the lifecycle stays the same as today: new secrets are created, old ones are expired/cleaned up, and the MachineDeployment is updated to reference the new secret.
I'll add a "Token secret and payload cache lifecycle" section to each workflow scenario covering this.
| return cg.doParse(configs, cg.haproxyRawConfig) | ||
| } | ||
|
|
||
| func (cg *ConfigGenerator) parseWithoutHaproxy(configs []corev1.ConfigMap) (string, error) { |
There was a problem hiding this comment.
might want to name this after rollout vs non rollout so it's extendable
There was a problem hiding this comment.
Agreed. I'll rename the table headers and descriptions to use "rollout" vs "non-rollout" terminology:
| Hash | Category | Inputs | Used for |
|---|---|---|---|
Hash() / HashWithoutVersion() |
Non-rollout | Full MCO config including HAProxy, pull secret name, additional trust bundle name, reconciled global config | User-data secret naming, payload generation |
RolloutHash() / RolloutHashWithoutVersion() |
Rollout | MCO config excluding HAProxy, pull secret name, additional trust bundle name, user-set global config (proxy spec, image spec — without computed defaults) | Rollout decisions |
This makes it clearer that the categories are extensible — if new management-side content is added in the future, it goes into the "non-rollout" hash only.
| ignition payload so new nodes always receive the latest configuration, but they | ||
| no longer trigger rollouts of existing nodes. |
There was a problem hiding this comment.
This doesn't sound correct. When only HAProxy changes:
- A new user-data secret IS generated (different name from Hash())
- But
propagateVersionAndTemplatedoes NOT updateMachineDeployment.Spec.Template.Spec.Bootstrap.DataSecretName(because rollout hash didn't change) - Therefore scale-up nodes created by the MachineDeployment will reference the OLD DataSecretName
- Scale-up nodes get stale management-side config, not "the full payload"
There was a problem hiding this comment.
You're right — the summary as written is incorrect. With the corrected design (discussed in the thread with @enxebre), when only management-side content changes, no new token/user-data secrets are created at all. The existing secrets remain valid, and the MachineDeployment continues to reference them. Scale-up nodes get the same config as existing nodes — which is the correct and intended behavior, since we're explicitly choosing not to roll out.
I'll update the summary to reflect this accurately: "Management-side changes do not trigger rollouts, and both existing and scale-up nodes retain the current configuration until the next spec-driven rollout."
| reference embedded in the ignition payload, even though data plane CRI-O | ||
| handles mirroring natively | ||
| ([OCPBUGS-86415](https://issues.redhat.com/browse/OCPBUGS-86415)). | ||
| - **`config.openshift.io` computed defaults**: The `globalConfigString()` |
There was a problem hiding this comment.
how would we differentiate default changes from user intent changes? I don't think this is addressed by this enhacement
There was a problem hiding this comment.
This is addressed in the Implementation Details section under "Two-hash architecture". The rolloutGlobalConfigString() function hashes only the user-set spec fields (HostedCluster.Spec.Configuration.Proxy, HostedCluster.Spec.Configuration.Image) without reconciling platform-specific defaults. The full reconciled config (with computed defaults like Status.NoProxy entries) continues to be used for payload generation via the non-rollout hash.
So the differentiation is: user intent = raw spec fields in the rollout hash; computed defaults = only in the non-rollout hash used for payload generation.
| updated to the current rollout hash. The nodes that were replaced have the | ||
| ignition payload that was generated at the start of the rollout — they do NOT | ||
| automatically pick up the mid-rollout management-side change. | ||
| 6. The `ConfigUpdatePending` condition may transition to `True` if the |
There was a problem hiding this comment.
can we detail how would this be implemented in the impl details section?
how does the controller decides what to set to current not rollout config, if there are two targets, the latest (written in the target annotation) and the one in flight (which is not stored in the annotation anymore because latest overrode that)?
There was a problem hiding this comment.
The controller doesn't need to track the in-flight target separately. The nodePoolCurrentRolloutConfig annotation only gets updated when MachineDeploymentComplete() returns true — at that point all machines match the latest template, so the controller writes the current computed rollout hash.
If a second spec-driven change arrives mid-rollout, CAPI handles the superseding: the new desired state updates the MachineDeployment template, and CAPI continues rolling until all machines match the newest template. The annotation simply records "what rollout hash are the nodes actually running" — it doesn't need to know what the previous in-flight target was, because by the time MachineDeploymentComplete() fires, the machines are already on the latest state.
This is already covered in the "Annotation lifecycle" table and the "Rollout decision change" section — the annotation is unchanged during a rollout and only updated on completion.
| (e.g., `Status.NoProxy` entries like network CIDRs, `169.254.169.254` for | ||
| AWS). If operator code changes these defaults, the serialized config changes | ||
| and triggers a rollout even though the user's spec did not change. | ||
| 2. Customer-facing spec changes (release upgrades, proxy config, user |
There was a problem hiding this comment.
Are changes like the HostedCluster's CR ImageContentSources included in this part as well?
There was a problem hiding this comment.
Yes — ImageContentSources is a user-set field on the HostedCluster spec, so changes to it are considered customer-facing and would continue to trigger rollouts. It flows into the ignition config and is included in both the rollout and non-rollout hashes.
That said, for ARO-HCP where this field has a dual nature (OCP image entries managed by the platform alongside user-provided entries), a platform-side change to the OCP image entries would also trigger a rollout since the field is treated as a single unit. If that becomes a concern, it could be addressed separately by splitting the platform-managed entries out of the rollout hash — but that's not in scope for this enhancement.
|
|
||
| #### Spec-driven rollout | ||
|
|
||
| 1. The cluster administrator updates `NodePool.spec.config` to add a |
There was a problem hiding this comment.
What about the HostedCluster.Spec.ImageContentSources changes? Same as #2042
There was a problem hiding this comment.
For aro-hcp this is two folds;
- OCP images are in this list; these are managed by ARO-HCP
- And some images comes from the user provided input
There was a problem hiding this comment.
Same answer as above — ImageContentSources is treated as a customer-facing spec field and changes to it trigger rollouts. The ARO-HCP dual-nature concern (platform-managed OCP image entries mixed with user-provided entries) is acknowledged but out of scope for this enhancement. If needed, splitting platform-managed entries out of the rollout hash could be a follow-up.
There was a problem hiding this comment.
The ARO-HCP dual-nature concern (platform-managed OCP image entries mixed with user-provided entries) is acknowledged but out of scope for this enhancement. If needed, splitting platform-managed entries out of the rollout hash could be a follow-up.
I agree. I see it happening as existing clusters will be upgrade to v5 but there are possibilities of solving that e.g do the change a part of the upgrade trigger so that it falls under that one major upgrade
| This enhancement decouples the rollout-triggering hash from management-side | ||
| implementation details by introducing a *rollout hash* derived only from | ||
| customer-facing spec inputs. Management-side changes do not trigger rollouts, | ||
| and both existing and scale-up nodes retain the current configuration until the |
There was a problem hiding this comment.
What is perhaps not super clear - are we omitting things from the hash which truly do not influence ignition, in which case this is fixing a bug, or which do influence it, so we are relaxing the behavior at a risk of correctness? If the latter, what is the risk and why is it worth the benefit?
There was a problem hiding this comment.
Good question — I've updated the Summary to make this distinction explicit. This is a deliberate relaxation, not a bug fix: the omitted inputs (HAProxy image, registry overrides, computed config defaults) do influence the ignition payload content. Excluding them from the rollout hash means existing and scale-up nodes will run with stale management-side configuration until the next spec-driven rollout.
The trade-off is acceptable because the previous management-side configuration remains functional — existing nodes continue to operate correctly — and the operational cost of unexpected fleet-wide node replacements far outweighs the benefit of immediate propagation of non-customer-facing changes.
| an expiration timestamp (2-hour TTL), and the previous user-data secret is | ||
| deleted (the MachineDeployment will be updated to reference the new one). | ||
| 3. The controller detects that `RolloutHashWithoutVersion()` differs from the | ||
| `nodePoolCurrentRolloutConfig` annotation. |
There was a problem hiding this comment.
Which CR will have this annotation the NodePool CR?
Followup question; is this an existing annotation or a new one? If a new one why an annotation over say a .status condition or something?
There was a problem hiding this comment.
Which CR will have this annotation the NodePool CR? Followup question; is this an existing annotation or a new one?
This is answered here https://github.com/csrwng/enhancements/blob/f1de573248003660bd274bc825d7d409b4208807/enhancements/hypershift/predictable-nodepool-rollout-control.md#api-extensions
If a new one why an annotation over say a .status condition or something?
I am curious to lean more on the difference between the two design choices. annotation vs .status.
In https://github.com/csrwng/enhancements/blob/f1de573248003660bd274bc825d7d409b4208807/enhancements/hypershift/predictable-nodepool-rollout-control.md#api-extensions I read that the annotation is controller-managed so a user wouldn't be able to set it, but I am keen to learn more on why an annotation was preferred here
There was a problem hiding this comment.
Yes, this annotation is on the NodePool CR — it's a new annotation described in the API Extensions section.
For why an annotation over .status: the annotation stores an opaque hash value that the controller uses internally to compare rollout state across reconciles. It's not meaningful to users — it's just a string like "a3f8b2c1". Status conditions are booleans with reason/message, designed for human-readable observability. An opaque comparison value doesn't fit that model.
The human-readable part is covered by the ConfigUpdatePending status condition, which tells operators whether management-side drift exists. So the two work together: the annotation is the internal mechanism, the condition is the external signal.
We also follow the existing pattern — nodePoolAnnotationCurrentConfig and nodePoolAnnotationCurrentConfigVersion are both annotations used the same way.
|
|
||
| #### Spec-driven rollout | ||
|
|
||
| 1. The cluster administrator updates `NodePool.spec.config` to add a |
There was a problem hiding this comment.
The HAProxy image can also be driven via the NodePoolHAProxyImageAnnotation annotation. Does this enhancement consider this a spec driven as well?
There was a problem hiding this comment.
In ARO-HCP we don't allow the user to BYO HAProxy image, so this technically a platform driven choice. But a question is worth asking on what hypershift considers this to be.
There was a problem hiding this comment.
The NodePoolHAProxyImageAnnotation is platform-driven, not spec-driven — it's the mechanism by which the operator injects the HAProxy image into the ignition payload. This enhancement explicitly treats it as management-side content excluded from the rollout hash. That's the core use case this enhancement addresses.
Changes to this annotation will update the non-rollout hash (affecting payload generation and user-data secret naming) but will NOT change the rollout hash, so no node replacement is triggered.
| 1. A `rolloutMcoRawConfig` field, computed by a new `parseWithoutHaproxy()` | ||
| method that calls the existing parsing logic with an empty haproxy config | ||
| string. | ||
| 2. A `rolloutGlobalConfig` field, computed by serializing only the user-set spec |
There was a problem hiding this comment.
I guess HC.spec.imageContentSources falls under this category as well?
There was a problem hiding this comment.
Yes — ImageContentSources feeds into the ignition config through the MCO config generation pipeline, so it's already captured in the "MCO config excluding HAProxy" input in the rollout hash. Changes to it will change the rollout hash and trigger a rollout.
| | Management-side change only | Annotation unchanged — no rollout | | ||
|
|
||
| The `currentRolloutConfigHash != ""` guard in the rollout decision ensures that | ||
| if the annotation is somehow absent (e.g., manually deleted), the controller |
There was a problem hiding this comment.
What happens when user sets it to some non empty value (Is it even possible or this action won't be possible to do by the user)?
There was a problem hiding this comment.
There's no admission webhook preventing it, so a user with direct access to NodePool objects (i.e., the service provider, not customers who own the hosted clusters — customers don't have access to management cluster resources like NodePools) could technically edit the annotation.
If they set it to a value that doesn't match the current computed rollout hash, the controller would see a mismatch and trigger a rollout on the next reconcile. If they set it to the current rollout hash, it's a no-op. If they delete it, the controller re-seeds it without triggering a rollout (the `currentRolloutConfigHash != """ guard handles this).
This is the same trust model as the existing nodePoolAnnotationCurrentConfig and nodePoolAnnotationCurrentConfigVersion annotations — service providers shouldn't touch them, but if they do, the worst case is an unnecessary rollout.
SudoBrendan
left a comment
SudoBrendan
left a comment
There was a problem hiding this comment.
Is forced updates by platforms on or off the table? If this has already been decided outside this doc, my apologies - tried to flag what such a thing might imply.
| 1. **In-place config refresh for existing nodes.** When management-side | ||
| configuration changes are excluded from triggering a Replace rollout, existing | ||
| worker nodes retain their previous configuration until the next spec-driven | ||
| rollout. Delivering non-disruptive configuration changes to existing nodes | ||
| without node replacement is tracked separately as | ||
| [OCPSTRAT-3299](https://issues.redhat.com/browse/OCPSTRAT-3299). |
There was a problem hiding this comment.
I'm still getting accustomed to all the nuances here operationally - but I could imagine a scenario where we need to push out a fleet-wide update related to vulnerabilities or similar (e.g. HAProxy has a critical CVE). Will we have a mechanism to force reconciliation if needed (either immediately or within a timeframe)? If I'm reading correctly - SREs would have no way to do this; customers are the only ones that can push updates out - if they don't do so, they don't get the fixes, potentially ever. Is this the known customer experience/acceptable risk?
There was a problem hiding this comment.
I don't believe we are on the hook for forcing a customer to change what is running in their subscription - we need to provide them the option to upgrade, but do not need to enforce that e.g. all those applications running in the customer's side are truly upgraded.
There was a problem hiding this comment.
Agreed with Steve's point — we're not on the hook for forcing customers to roll out changes in their subscription.
That said, this enhancement does provide the visibility layer: the ConfigUpdatePending condition tells SREs exactly which NodePools have pending management-side changes that haven't been rolled out yet. This can be used to build fleet-level dashboards showing which clusters are running stale management-side config.
For the force-rollout mechanism: that's out of scope for this enhancement but could be a follow-up. The architecture supports it — a service provider with access to NodePool objects could trigger a rollout by making a no-op spec change (e.g., adding and removing a label on a MachineConfig). A more formal mechanism (e.g., an annotation that explicitly requests a rollout) could be designed separately if needed.
Delivering management-side changes to existing nodes without full node replacement is tracked as OCPSTRAT-3299.
| - `ConfigUpdatePending` condition is documented and surfaced in monitoring | ||
| dashboards. |
There was a problem hiding this comment.
I assume monitoring dashboards implies that we'll have a metric to query? Is it worth defining in this doc?
Operationally, this would be invaluable for us to see what percentage of clusters in our fleet are pending an update, and which have completed. It would be even better if there was some way to differentiate distinct updates we push in this metric - I can almost guarantee we'll have some customers that don't make an update call for months or even years (e.g. If we could see they got update foo but not bar or biz). This would also be incredible if we do have that "force" button for critical things that must go out immediately - because we can see/prove with a dashboard that "yes, the fleet got the update - every single cluster" or "cluster ABC missed the update, we have to figure out why".
There was a problem hiding this comment.
The ConfigUpdatePending condition is the metric source — kube-state-metrics already exposes NodePool conditions as Prometheus metrics, so fleet-level queries like "what percentage of NodePools have pending management-side updates" are immediately available without defining new custom metrics.
For differentiating distinct updates: the condition's message field describes the drift source (e.g., ManagementConfigDrift), but it doesn't currently encode which specific update is pending. Adding a more granular breakdown (e.g., which hash changed, or what content differs) could be a follow-up if the basic condition isn't sufficient for operational needs.
Specific dashboard and alerting design is listed as a GA graduation criterion — we'll define those details as part of the production rollout rather than in the enhancement itself.
| The new `ConfigUpdatePending` condition is a standard NodePool status condition | ||
| and does not affect API throughput or availability. It is updated as part of the | ||
| existing NodePool reconciliation loop. |
There was a problem hiding this comment.
note: if we do add a "force" button for SRE - we'd likely want to differentiate what kind of rollout happened (customer or platform initiated)
There was a problem hiding this comment.
Good note. The ConfigUpdatePending condition already provides some of this distinction — if it was True before a rollout and transitions to False after, you know the rollout picked up pending management-side changes alongside the spec-driven change.
Explicit tracking of "who initiated this rollout" (customer vs platform) would be a follow-up concern if a force-rollout mechanism is added. Noted for future design.
| and does not affect API throughput or availability. It is updated as part of the | ||
| existing NodePool reconciliation loop. | ||
|
|
||
| ### Failure Modes |
There was a problem hiding this comment.
what happens when customers have PDBs that block rollouts? Is this worth documenting here - how it may/may not impact the annotation lifecycle?
There was a problem hiding this comment.
PDBs blocking rollouts is existing CAPI behavior unchanged by this enhancement. If a PDB blocks machine draining, the MachineDeployment rollout stalls — MachineDeploymentComplete() never returns true, so the nodePoolCurrentRolloutConfig annotation stays at the old value and the UpdatingConfig condition remains True.
The rollout eventually times out per CAPI's progressDeadlineSeconds. This enhancement doesn't change any of that — the annotation lifecycle is purely driven by the existing completion signals. No special PDB handling is needed.
Introduces a two-hash architecture in the NodePool controller to decouple rollout decisions from management-side configuration changes. A new "rollout hash" derived only from customer-facing spec inputs (user MachineConfigs, release version, pull secret, trust bundle, user-set proxy/image config) determines whether to trigger Replace or InPlace rollouts. Management-side changes (HAProxy image bumps, registry overrides, config.openshift.io computed defaults) no longer trigger rollouts. Tracking: CNTRLPLANE-3631
|
@csrwng: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
7 participants
Summary
Enhancement proposal for OCPSTRAT-3298 — Predictable NodePool Rollout Control for Hosted Control Planes.
Today, the HyperShift NodePool controller uses a single hash over the entire rendered ignition config to drive rollout decisions. Any change — including automated management-side image digest bumps (e.g., HAProxy) — triggers a full Replace rollout of all worker nodes. This has caused production incidents.
This enhancement proposes:
nodePoolCurrentRolloutConfigannotation with safe first-reconcile seeding for existing NodePoolsConfigUpdatePendingcondition for observability into management-side configuration driftJira
Test plan
🤖 Generated with Claude Code