Kubernetes 1.29 Major Themes

[fsmunoz]

This is a discussion to discuss Major Themes for the v1.29 release.

Here is the discussion thread from the last release cycle.

cc @kubernetes/release-team-leads
cc @fykaa @mengjiao-liu @muddapu @rashansmith

[sanposhiho]

From SIG-Scheduling:

• Per-plugin callback functions for accurate requeueing in kube-scheduler enhancements#4247
  • The throughput of the scheduler is our eternal challenge. This QueueingHint feature brings a new possibility to optimize the efficiency of requeueing, which could reduce useless scheduling retries significantly. We're actively working on this, and some stable in-tree plugins will have supported it in v1.29.
• Introduce MatchLabelKeys to Pod Affinity and Pod Anti Affinity enhancements#3633
  • One enhancement will be introduced in PodAffinity/PodAntiAffinity as alpha. It will increase the accuracy of calculation during rolling updates.
• ReadWriteOncePod PersistentVolume Access Mode enhancements#2485
  • ReadWriteOncePod access mode for PV is moving towards GA.
• Decouple TaintManager from NodeLifecycleController enhancements#3902
  • It's marked as beta, but actually a completely new change. (It skips the alpha.)
  • As title describes, it's to decouple TaintManager that performs taint-based pod eviction from NodeLifecycleController and make them two separate controllers: NodeLifecycleController to add taints to unhealthy nodes and TaintManager to perform pod deletion on nodes tainted with NoExecute effect.

@kubernetes/sig-scheduling-approvers Please follow up if anything missing.

[alculquicondor]

That's a great summary, thank you @sanposhiho

[justankiit]

That is a great Summary @sanposhiho

[kerthcet]

Thanks @sanposhiho for the effort.

[xing-yang]

From SIG-Storage:

• ReadWriteOncePod Persistent Volume Access Mode feature moved to GA in v1.29. This feature introduces a new ReadWriteOncePod access mode for PersistentVolumes that restricts access to a single pod on a single node at a time.
• NodeExpandSecret feature moved to GA in v1.29. This feature adds NodeExpandSecret to the CSI persistent volume source and enables the CSI client to send it out as part of the nodeExpandVolume request to the CSI drivers.
• The VolumeAttributesClass Alpha feature was introduced in v1.29. This feature proposes an extension to the Kubernetes Persistent Volume API to allow users to change volume attributes after provisioning (for example, for IOPS or throughput).

cc @saad-ali @msau42 @jsafrane

[carlory]

In 1.29, the VolumeAttributesClass feature maybe only contain the API changes and its functions are not implemented yet. So, I'm not sure whether we should add an announcement in the major themes.

cc @msau42 @sunnylovestiramisu

[xing-yang]

Implementation is in external-provisioner and external-resizer which will be released async after K8s 1.29 release.

[aojea]

From SIG-Network:

• nftables kube-proxy backend Alpha feature introduced in v1.29. This features adds a new backend to kube-proxy based on nftables, since kernel community is focused mainly in nftables and practically stopped the iptables development
• Multiple Service CIDRs. Alpha feature introduced in v1.29. This features allow cluster administrators to dynamically resize the service IP ranges assigned to their clusters to deal with problems like IP exhaustion or IP renumbering.

cc: @danwinship @thockin @shaneutt

[aravindhp]

From SIG-Windows:

• Support image pull per runtime class enhancements#4216
  • Alpha feature introduced in v1.29
• In-Place Update of Pod Resources enhancements#1287
  • Windows support going Alpha in v1.29

[elmiko]

From SIG-Cloud Provider:

• [KEP-2395] Update 2395-removing-in-tree-cloud-providers to beta status enhancements#4190
  • feature moved to beta status, users need to be aware that the feature gates DisableCloudProviders and DisableKubeletCloudCredentialProvider will now default to true. This means that users on Azure, GCE, and vSphere who are not using external cloud controller managers will need to set these feature gates to false, or migrate to external CCMs.

⚠️ This has the potential to break users who are not prepared for the change in feature gates.

cc @bridgetkromhout @andrewsykim @cheftako

[logicalhan]

From SIG-Instrumentation:

• Graduate kubelet resource metrics to GA kubernetes#116897
  • Kubelet resource metrics are graduated to GA
• promote component SLIs to GA; remove feature gates for component slis kubernetes#120574
  • Component SLI metrics are graduated to GA
• contextual logging enhancements#3077
  • kube-scheduler was converted to contextual logging

[mrunalp]

From SIG-Node:

• Promote SidecarContainers feature to beta kubernetes#121579 Sidecar containers beta
• Kubelet limit of Parallel Image Pulls enhancements#3673 Max image parallel pulls beta
• KEP-127: Update PSS based on feature gate kubernetes#118760 User namespaces pod security alpha
• Support image pull per runtime class enhancements#4216 Image pull per runtime class alpha
• Kubelet support for Split Image Filesystem enhancements#4191 Split filesystem alpha
• kubelet image GC after a maximum age enhancements#4210 Kubelet image GC after maximum age alpha

[SergeyKanzhelev]

We also want to highlight that Evente PLEG beta feature that is disabled by default have a critical bug: kubernetes/kubernetes#121349 (comment) that is not resolved yet. We recommend to use extra caution when enabling this feature.

[liggitt]

We also want to highlight that Evente PLEG beta feature that is disabled by default have a critical bug: kubernetes/kubernetes#121349 (comment) that is not resolved yet. We recommend to use extra caution when enabling this feature.

That's a good candidate for the "Known Issues" section of the release notes

[SergeyKanzhelev]

Is there a mechanism to include it there?

[liggitt]

add to kubernetes/kubernetes#121886

[pacoxu]

kubernetes/enhancements#3673

I added a new e2e in kubernetes/kubernetes#121604 which is not merged yet. The adding e2e is a beta criterion for this feature. However, there is no FG for this KEP.

[ritazh]

From SIG-Auth:

• [KMSv2] promote KMSv2 and KMSv2KDF to GA kubernetes#121485
  • 3299 KMS v2 encryption at rest improvement graduates to GA in 1.29
• LegacyServiceAccountTokenCleanUp beta kubernetes#120682
  • 2799 Reduce secret based SA token, LegacyServiceAccountTokenCleanUp going to Beta in 1.29
• [StructuredAuthorizationConfig] - Implement AuthorizationConfiguration kubernetes#118872
  • 3221 Structured Authorization Configuration going Alpha in 1.29
• [StructuredAuthenticationConfig] Add feature flag and wire up --authentication-config flag kubernetes#119142
  • 3331 Structured OIDC Configuration going Alpha in 1.29
• Implement ClusterTrustBundlePEM projected volume kubernetes#113374
  • 3257 Cluster Trust Bundles going Alpha in 1.29
• Including JTI & node reference in issued service account tokens (kep 4193) kubernetes#120780
  • 4193 Bound service account token improvements going Alpha in 1.29

cc @liggitt @enj @aramase @ahmedtd @munnerz @nilekhc @palnabarun @yt2985

[liggitt]

If we wanted a focused list, the KMS GA and token cleanup beta are the ones I'd highlight. The alpha stuff is interesting as a preview of what's coming but is admin-focused or not enabled by default.

[ameukam]

From SIG Release:

• apt.kubernetes.io & yum.kubernetes.io are going away. See https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/

[BenTheElder]

Can we also clarify the backing kubernetes packages on packages.cloud.google.com as mentioned in kubernetes/website#44109 (comment)

"apt.kubernetes.io & yum.kubernetes.io AKA packages.cloud.google.com legacy Kubernetes packages are going away, to be replaced by pkgs.k8s.io. See: https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/"

[mrbobbytables]

+1 from me

[krol3]

Hi Team, here Carol from the Communication 1.29, we will like to choose the top 3 major themes for the webinar 1.29. What do you think? @kubernetes/sig-release-leads cc: @Priyankasaggu11929 @kubernetes/release-team-comms

• KEP-2395 Update 2395-removing-in-tree-cloud-providers to beta status
• KEP-3299 KMS v2 Improvements #3299
• KEP-1287 In-Place Update of Pod Resources Cut 1.17.13 release #1287

[elmiko]

+1 for kep-2395 being in there