Kubernetes

- Update to kubernetes 1.6 spec 169

API Change
- 'ACTION REQUIRED: The Dynamic Resource Allocation (DRA) driver's DaemonSet
must be deployed with a service account that enables writing ResourceSlice
and reading ResourceClaim objects.'
([kubernetes/kubernetes125163](https://github.com/kubernetes/kubernetes/pull/125163), [pohly](https://github.com/pohly)) [SIG Auth, Node and Testing]
- Add UserNamespaces field to NodeRuntimeHandlerFeatures ([kubernetes/kubernetes126034](https://github.com/kubernetes/kubernetes/pull/126034), [sohankunkerkar](https://github.com/sohankunkerkar)) [SIG API Machinery, Apps and Node]
- Added Coordinated Leader Election as Alpha under the `CoordinatedLeaderElection` feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. ([kubernetes/kubernetes124012](https://github.com/kubernetes/kubernetes/pull/124012), [Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]
- Added a `.status.features.supplementalGroupsPolicy` field to Nodes. The field is true when the feature is implemented in the CRI implementation (KEP-3619). ([kubernetes/kubernetes125470](https://github.com/kubernetes/kubernetes/pull/125470), [everpeace](https://github.com/everpeace)) [SIG API Machinery, Apps, Node and Testing]
- Added an `allocatedResourcesStatus` to each container status to indicate the health status of devices exposed by the device plugin. ([kubernetes/kubernetes126243](https://github.com/kubernetes/kubernetes/pull/126243), [SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG API Machinery, Apps, Node and Testing]
- Added support to the kube-proxy nodePortAddresses / --nodeport-addresses option to
accept the value "primary", meaning to only listen for NodePort connections
on the node's primary IPv4 and/or IPv6 address (according to the Node object).
This is strongly recommended, if you were not previously using
--nodeport-addresses, to avoid surprising behavior.
(This behavior is enabled by default with the nftables backend; you would
need to explicitly request `--nodeport-addresses 0.0.0.0/0,::/0` there to get
the traditional "listen on all interfaces" behavior.) ([kubernetes/kubernetes123105](https://github.com/kubernetes/kubernetes/pull/123105), [danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Windows]
- Added the feature gates `StrictCostEnforcementForVAP` and `StrictCostEnforcementForWebhooks` to enforce the strict cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. ([kubernetes/kubernetes124675](https://github.com/kubernetes/kubernetes/pull/124675), [cici37](https://github.com/cici37)) [SIG API Machinery, Auth, Node and Testing]
- Changed how the API server handles updates to `.spec.defaultBackend` of Ingress objects.
Server-side apply now considers `.spec.defaultBackend` to be an atomic struct. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact; for controllers that want to change the default backend port from number to name (or vice-versa), this makes it easier. ([kubernetes/kubernetes126207](https://github.com/kubernetes/kubernetes/pull/126207), [thockin](https://github.com/thockin)) [SIG API Machinery]
- Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. ([kubernetes/kubernetes120696](https://github.com/kubernetes/kubernetes/pull/120696), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]
- CustomResourceDefinition objects created with non-empty `caBundle` fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid `caBundle` is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid `caBundle` field to an invalid `caBundle` field, because this breaks serving of the existing CustomResourceDefinition. ([kubernetes/kubernetes124061](https://github.com/kubernetes/kubernetes/pull/124061), [Jefftree](https://github.com/Jefftree)) [SIG API Machinery]
- Dynamic Resource Allocation (DRA): Added a feature so the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. ([kubernetes/kubernetes120611](https://github.com/kubernetes/kubernetes/pull/120611), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]
- Dynamic Resource Allocation (DRA): client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. ([kubernetes/kubernetes124075](https://github.com/kubernetes/kubernetes/pull/124075), [pohly](https://github.com/pohly))
- Dynamic Resource Allocation (DRA): in the `pod.spec.recourceClaims` array, the `source` indirection is no longer necessary. Instead of e.g. `source: resourceClaimTemplateName: my-template`, one can write `resourceClaimTemplateName: my-template`. ([kubernetes/kubernetes125116](https://github.com/kubernetes/kubernetes/pull/125116), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- Enhanced the Dynamic Resource Allocation (DRA) with an updated version of the resource.k8s.io API group. The primary user-facing type remains the ResourceClaim, however significant changes have been made, resulting in the new version, v1alpha3, which is not compatible with the previous version. ([kubernetes/kubernetes125488](https://github.com/kubernetes/kubernetes/pull/125488), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]
- Fixed a 1.30.0 regression in OpenAPI descriptions of the `imagePullSecrets` and
`hostAliases` fields to mark the fields used as keys in those lists as either defaulted
or required. ([kubernetes/kubernetes124553](https://github.com/kubernetes/kubernetes/pull/124553), [pmalek](https://github.com/pmalek))
- Fixed a 1.30.0 regression in openapi descriptions of `PodIP.IP` and `HostIP.IP` fields to mark the fields used as keys in those lists as required. ([kubernetes/kubernetes126057](https://github.com/kubernetes/kubernetes/pull/126057), [thockin](https://github.com/thockin))
- Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an `items` field. ([kubernetes/kubernetes124568](https://github.com/kubernetes/kubernetes/pull/124568), [xyz-li](https://github.com/xyz-li)) [SIG API Machinery]
- Fixed a deep copy issue when retrieving the controller reference. ([kubernetes/kubernetes124116](https://github.com/kubernetes/kubernetes/pull/124116), [HiranmoyChowdhury](https://github.com/HiranmoyChowdhury)) [SIG API Machinery and Release]
- Fixed code-generator client-gen to work with `api/v1`-like package structure. ([kubernetes/kubernetes125162](https://github.com/kubernetes/kubernetes/pull/125162), [sttts](https://github.com/sttts)) [SIG API Machinery and Apps]
- Fixed incorrect "v1 Binding is deprecated in v1.6+" warning in kube-scheduler log. ([kubernetes/kubernetes125540](https://github.com/kubernetes/kubernetes/pull/125540), [pohly](https://github.com/pohly)) [SIG API Machinery]
- Fixed the comment for the Job's managedBy field. ([kubernetes/kubernetes124793](https://github.com/kubernetes/kubernetes/pull/124793), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Fixed the documentation for the default value of the `procMount` entry in `securityContext` within a Pod.
The documentation was previously using the name of the internal variable `DefaultProcMount`, rather than the actual value, "Default". ([kubernetes/kubernetes125782](https://github.com/kubernetes/kubernetes/pull/125782), [aborrero](https://github.com/aborrero)) [SIG Apps and Node]
- Graduate PodDisruptionConditions to GA and lock ([kubernetes/kubernetes125461](https://github.com/kubernetes/kubernetes/pull/125461), [mimowo](https://github.com/mimowo)) [SIG Apps, Node, Scheduling and Testing]
- Graduated MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta. ([kubernetes/kubernetes123638](https://github.com/kubernetes/kubernetes/pull/123638), [sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Scheduling and Testing]
- Graduated `JobPodFailurePolicy` to GA and locked it to it's default. ([kubernetes/kubernetes125442](https://github.com/kubernetes/kubernetes/pull/125442), [mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps, Scheduling and Testing]
- Graduated the Job `successPolicy` field to beta.

The new reason label, "SuccessPolicy" and "CompletionsReached" are added to the "jobs_finished_total" metric.
Additionally, if you enable the `JobSuccessPolicy` feature gate, the Job gets "CompletionsReached" reason for the "SuccessCriteriaMet" and "Complete" condition type
when the number of succeeded Job Pods (`.status.succeeded`) reached the desired completions (`.spec.completions`). ([kubernetes/kubernetes126067](https://github.com/kubernetes/kubernetes/pull/126067), [tenzen-y](https://github.com/tenzen-y)) [SIG API Machinery, Apps and Testing]
- Graduated the `DisableNodeKubeProxyVersion` feature gate to beta. By default, the kubelet no longer attempts to set the `.status.kubeProxyVersion` field for its associated Node. ([kubernetes/kubernetes123845](https://github.com/kubernetes/kubernetes/pull/123845), [HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Cloud Provider, Network, Node and Testing]
- Improved scheduling performance when many nodes, and prefilter returned 1-2 nodes (e.g. daemonset)

For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status. ([kubernetes/kubernetes125197](https://github.com/kubernetes/kubernetes/pull/125197), [gabesaba](https://github.com/gabesaba))
- Introduced a new boolean kubelet flag `--fail-cgroupv1`. ([kubernetes/kubernetes126031](https://github.com/kubernetes/kubernetes/pull/126031), [harche](https://github.com/harche)) [SIG API Machinery and Node]
- K8s.io/apimachinery/pkg/util/runtime: Added support for new calls to handle panics and errors in the context where they occur. `PanicHandlers` and `ErrorHandlers` now must accept a context parameter for that. Log output is structured instead of unstructured. ([kubernetes/kubernetes121970](https://github.com/kubernetes/kubernetes/pull/121970), [pohly](https://github.com/pohly)) [SIG API Machinery and Instrumentation]
- KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite. ([kubernetes/kubernetes122047](https://github.com/kubernetes/kubernetes/pull/122047), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Instrumentation and Testing]
- Kube-apiserver: Added Alpha features to allow API server authz to check the context of requests:
- The `AuthorizeWithSelectors` feature gate enables including field and label selector information from requests in webhook authorization calls.
- The `AuthorizeNodeWithSelectors` feature gate changes node authorizer behavior to limit requests from node API clients, so that each Node can only get / list / watch its own Node API object, and can also only get / list / watch Pod API objects bound to that node. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or obtain broader read access independent of the node authorizer. ([kubernetes/kubernetes125571](https://github.com/kubernetes/kubernetes/pull/125571), [liggitt](https://github.com/liggitt)) [SIG API Machinery, Auth, Node, Scheduling and Testing]
- Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the `data` field. ([kubernetes/kubernetes125549](https://github.com/kubernetes/kubernetes/pull/125549), [liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps]
- Kube-apiserver: the `--encryption-provider-config` file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When `--encryption-provider-config-automatic-reload` is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. ([kubernetes/kubernetes124912](https://github.com/kubernetes/kubernetes/pull/124912), [enj](https://github.com/enj)) [SIG API Machinery and Auth]
- Kube-controller-manager: the `horizontal-pod-autoscaler-upscale-delay` and `horizontal-pod-autoscaler-downscale-delay` flags have been removed (deprecated and non-functional since v1.12). ([kubernetes/kubernetes124948](https://github.com/kubernetes/kubernetes/pull/124948), [SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Apps and Autoscaling]
- Made kube-proxy Windows service control manager integration (`--windows-service`) configurable in v1alpha1 component configuration via `windowsRunAsService` field. ([kubernetes/kubernetes126072](https://github.com/kubernetes/kubernetes/pull/126072), [aroradaman](https://github.com/aroradaman)) [SIG Network and Scalability]
- PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. ([kubernetes/kubernetes124969](https://github.com/kubernetes/kubernetes/pull/124969), [RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Storage and Testing]
- Promoted `LocalStorageCapacityIsolation` to beta; the behaviour is enabled by default. Within the kubelet, storage capacity isolation is active if the feature gate is enabled and the specific Pod is using a user namespace. ([kubernetes/kubernetes126014](https://github.com/kubernetes/kubernetes/pull/126014), [PannagaRao](https://github.com/PannagaRao)) [SIG Apps, Autoscaling, Node, Storage and Testing]
- Promoted `StatefulSetStartOrdinal` to stable. This means `--feature-gates=StatefulSetStartOrdinal=true` are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation. ([kubernetes/kubernetes125374](https://github.com/kubernetes/kubernetes/pull/125374), [pwschuurman](https://github.com/pwschuurman)) [SIG API Machinery, Apps and Testing]
- Promoted feature-gate `VolumeAttributesClass` to beta (disabled by default). Users need to enable the feature gate and the `storage.k8s.io/v1beta1` API group to use this feature.
Promoted the VolumeAttributesClass API to beta. ([kubernetes/kubernetes126145](https://github.com/kubernetes/kubernetes/pull/126145), [carlory](https://github.com/carlory)) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]
- Removed deprecated command flags --volume-host-cidr-denylist
and --volume-host-allow-local-loopback from kube-controller-manager.
([kubernetes/kubernetes124017](https://github.com/kubernetes/kubernetes/pull/124017), [carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Cloud Provider and Storage]
- Removed feature gate `CustomResourceValidationExpressions`. ([kubernetes/kubernetes126136](https://github.com/kubernetes/kubernetes/pull/126136), [cici37](https://github.com/cici37)) [SIG API Machinery, Cloud Provider and Testing]
- Reverted a [change](https://github.com/kubernetes/kubernetes/pull/123513) where `ConsistentListFromCache` was moved to beta and enabled by default. ([kubernetes/kubernetes#126139](https://github.com/kubernetes/kubernetes/pull/126139), [enj](https://github.com/enj))
- Revised the Pod API with Alpha support for volumes derived from OCI artifacts. This feature is behind the `ImageVolume` feature gate. ([kubernetes/kubernetes125660](https://github.com/kubernetes/kubernetes/pull/125660), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Apps and Node]
- Supported fine-grained supplemental groups policy (KEP-3619), which enabled
fine-grained control for supplementary groups in the first container processes.
This allows you to choose whether to include groups defined in the container image (/etc/groups)
for the container's primary UID or not. ([kubernetes/kubernetes117842](https://github.com/kubernetes/kubernetes/pull/117842), [everpeace](https://github.com/everpeace)) [SIG API Machinery, Apps and Node]
- The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
of the nft command-line, and kernel 5.13 or later. (For testing/development
purposes, you can use older kernels, as far back as 5.4, if you set the
`nftables.skipKernelVersionCheck` option in the kube-proxy config, but this is not
recommended in production since it may cause problems with other nftables
users on the system.) ([kubernetes/kubernetes124152](https://github.com/kubernetes/kubernetes/pull/124152), [danwinship](https://github.com/danwinship)) [SIG Network]
- To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. ([kubernetes/kubernetes126188](https://github.com/kubernetes/kubernetes/pull/126188), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Updated the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. ([kubernetes/kubernetes125021](https://github.com/kubernetes/kubernetes/pull/125021), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]
- Use omitempty for optional Job Pod Failure Policy fields. ([kubernetes/kubernetes126046](https://github.com/kubernetes/kubernetes/pull/126046), [mimowo](https://github.com/mimowo))
- User can choose a different static policy option `SpreadPhysicalCPUsPreferredOption` to spread cpus across physical cpus for some specific applications ([kubernetes/kubernetes123733](https://github.com/kubernetes/kubernetes/pull/123733), [Jeffwan](https://github.com/Jeffwan)) [SIG Node]
- When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. ([kubernetes/kubernetes124917](https://github.com/kubernetes/kubernetes/pull/124917), [vinayakankugoyal](https://github.com/vinayakankugoyal)) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
- Move ConsistentListFromCache feature flag to Beta and enable it by default ([kubernetes/kubernetes126469](https://github.com/kubernetes/kubernetes/pull/126469), [serathius](https://github.com/serathius)) [SIG API Machinery]
- Add Coordinated Leader Election as alpha under the CoordinatedLeaderElection feature gate. With the feature enabled, the control plane can use LeaseCandidate objects (coordination.k8s.io/v1alpha1 API group) to participate in a leader election and let the kube-apiserver select the best instance according to some strategy. ([kubernetes/kubernetes124012](https://github.com/kubernetes/kubernetes/pull/124012), [Jefftree](https://github.com/Jefftree)) [SIG API Machinery, Apps, Auth, Cloud Provider, Etcd, Node, Release, Scheduling and Testing]
- Add an AllocatedResourcesStatus to each container status to indicate the health status of devices exposed by the device plugin. ([kubernetes/kubernetes126243](https://github.com/kubernetes/kubernetes/pull/126243), [SergeyKanzhelev](https://github.com/SergeyKanzhelev)) [SIG API Machinery, Apps, Node and Testing]
- Added Node.Status.Features.SupplementalGroupsPolicy field which is set to true when the feature is implemented in the CRI implementation (KEP-3619) ([kubernetes/kubernetes125470](https://github.com/kubernetes/kubernetes/pull/125470), [everpeace](https://github.com/everpeace)) [SIG API Machinery, Apps, Node and Testing]
- CustomResourceDefinition objects created with non-empty `caBundle` fields which are invalid or do not contain any certificates will not appear in discovery or serve endpoints until a valid `caBundle` is provided. Updates to CustomResourceDefinition are no longer allowed to transition a valid `caBundle` field to an invalid `caBundle` field. ([kubernetes/kubernetes124061](https://github.com/kubernetes/kubernetes/pull/124061), [Jefftree](https://github.com/Jefftree)) [SIG API Machinery]
- DRA: The DRA driver's daemonset must be deployed with a service account that enables writing ResourceSlice and reading ResourceClaim objects. ([kubernetes/kubernetes125163](https://github.com/kubernetes/kubernetes/pull/125163), [pohly](https://github.com/pohly)) [SIG Auth, Node and Testing]
- DRA: new API and several new features ([kubernetes/kubernetes125488](https://github.com/kubernetes/kubernetes/pull/125488), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Etcd, Node, Release, Scheduling, Storage and Testing]
- DRA: the number of ResourceClaim objects can be limited per namespace and by the number of devices requested through a specific class via the v1.ResourceQuota mechanism. ([kubernetes/kubernetes120611](https://github.com/kubernetes/kubernetes/pull/120611), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Etcd, Node, Release, Scheduling and Testing]
- Fix the documentation for the default value of the procMount entry in the pod securityContext.
The documentation was previously using the name of the internal variable 'DefaultProcMount' rather than the actual value 'Default'. ([kubernetes/kubernetes125782](https://github.com/kubernetes/kubernetes/pull/125782), [aborrero](https://github.com/aborrero)) [SIG Apps and Node]
- Fixed a bug in the API server where empty collections of ValidatingAdmissionPolicies did not have an `items` field. ([kubernetes/kubernetes124568](https://github.com/kubernetes/kubernetes/pull/124568), [xyz-li](https://github.com/xyz-li)) [SIG API Machinery]
- Graduate the Job SuccessPolicy to Beta.

The new reason label, "SuccessPolicy" and "CompletionsReached" are added to the "jobs_finished_total" metric.
Additionally, If we enable the "JobSuccessPolicy" feature gate, the Job gets "CompletionsReached" reason for the "SuccessCriteriaMet" and "Complete" condition type
when the number of succeeded Job Pods (".status.succeeded") reached the desired completions (".spec.completions"). ([kubernetes/kubernetes126067](https://github.com/kubernetes/kubernetes/pull/126067), [tenzen-y](https://github.com/tenzen-y)) [SIG API Machinery, Apps and Testing]
- Introduce a new boolean kubelet flag --fail-cgroupv1 ([kubernetes/kubernetes126031](https://github.com/kubernetes/kubernetes/pull/126031), [harche](https://github.com/harche)) [SIG API Machinery and Node]
- Kube-apiserver: adds an alpha AuthorizeWithSelectors feature that includes field and label selector information from requests in webhook authorization calls; adds an alpha AuthorizeNodeWithSelectors feature that makes the node authorizer limit requests from node API clients to get / list / watch its own Node API object, and to get / list / watch its own Pod API objects. Clients using kubelet credentials to read other nodes or unrelated pods must change their authentication credentials (recommended), adjust their usage, or grant broader read access independent of the node authorizer. ([kubernetes/kubernetes125571](https://github.com/kubernetes/kubernetes/pull/125571), [liggitt](https://github.com/liggitt)) [SIG API Machinery, Auth, Node, Scheduling and Testing]
- Kube-proxy Windows service control manager integration(--windows-service) is now configurable in v1alpha1 component configuration via `WindowsRunAsService` field ([kubernetes/kubernetes126072](https://github.com/kubernetes/kubernetes/pull/126072), [aroradaman](https://github.com/aroradaman)) [SIG Network and Scalability]
- Promote LocalStorageCapacityIsolation to beta and enable if user namespace is enabled for the pod ([kubernetes/kubernetes126014](https://github.com/kubernetes/kubernetes/pull/126014), [PannagaRao](https://github.com/PannagaRao)) [SIG Apps, Autoscaling, Node, Storage and Testing]
- Promote StatefulSetStartOrdinal to stable. This means `--feature-gates=StatefulSetStartOrdinal=true` are not needed on kube-apiserver and kube-controller-manager binaries and they'll be removed soon following policy at https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecation ([kubernetes/kubernetes125374](https://github.com/kubernetes/kubernetes/pull/125374), [pwschuurman](https://github.com/pwschuurman)) [SIG API Machinery, Apps and Testing]
- Promoted feature-gate `VolumeAttributesClass` to beta (disabled by default). Users need to enable the feature gate and the storage v1beta1 group to use this new feature.
- Promoted API `VolumeAttributesClass` and `VolumeAttributesClassList` to `storage.k8s.io/v1beta1`. ([kubernetes/kubernetes126145](https://github.com/kubernetes/kubernetes/pull/126145), [carlory](https://github.com/carlory)) [SIG API Machinery, Apps, CLI, Etcd, Storage and Testing]
- Removed feature gate `CustomResourceValidationExpressions`. ([kubernetes/kubernetes126136](https://github.com/kubernetes/kubernetes/pull/126136), [cici37](https://github.com/cici37)) [SIG API Machinery, Cloud Provider and Testing]
- Revert "Move ConsistentListFromCache feature flag to Beta and enable it by default" ([kubernetes/kubernetes126139](https://github.com/kubernetes/kubernetes/pull/126139), [enj](https://github.com/enj)) [SIG API Machinery]
- Revised the Pod API with alpha support for volumes derived from OCI artefacts.
This feature is behind the `ImageVolume` feature gate. ([kubernetes/kubernetes125660](https://github.com/kubernetes/kubernetes/pull/125660), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Apps and Node]
- The Ingress.spec.defaultBackend is now considered an atomic struct for the purposes of server-side-apply. This means that any field-owner who sets values in that struct (they are mutually exclusive) owns the whole struct. For almost all users this change has no impact. For controllers which want to change port from number to name (or vice-versa), this makes it easier. ([kubernetes/kubernetes126207](https://github.com/kubernetes/kubernetes/pull/126207), [thockin](https://github.com/thockin)) [SIG API Machinery]
- To enhance usability and developer experience, CRD validation rules now support direct use of (CEL) reserved keywords as field names in object validation expressions for existing expressions in storage, will fully support runtime in next release for compatibility concern. ([kubernetes/kubernetes126188](https://github.com/kubernetes/kubernetes/pull/126188), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Add UserNamespaces field to NodeRuntimeHandlerFeatures ([kubernetes/kubernetes126034](https://github.com/kubernetes/kubernetes/pull/126034), [sohankunkerkar](https://github.com/sohankunkerkar)) [SIG API Machinery, Apps and Node]
- Fixes a 1.30.0 regression in openapi descriptions of PodIP.IP and HostIP.IP fields to mark the fields used as keys in those lists as required. ([kubernetes/kubernetes126057](https://github.com/kubernetes/kubernetes/pull/126057), [thockin](https://github.com/thockin)) [SIG API Machinery]
- Graduate JobPodFailurePolicy to GA and lock ([kubernetes/kubernetes125442](https://github.com/kubernetes/kubernetes/pull/125442), [mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps, Scheduling and Testing]
- Graduate PodDisruptionConditions to GA and lock ([kubernetes/kubernetes125461](https://github.com/kubernetes/kubernetes/pull/125461), [mimowo](https://github.com/mimowo)) [SIG Apps, Node, Scheduling and Testing]
- PersistentVolumeLastPhaseTransitionTime feature is stable and enabled by default. ([kubernetes/kubernetes124969](https://github.com/kubernetes/kubernetes/pull/124969), [RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Storage and Testing]
- The (alpha) nftables mode of kube-proxy now requires version 1.0.1 or later
of the nft command-line, and kernel 5.13 or later. (For testing/development
purposes, you can use older kernels, as far back as 5.4, if you set the
`nftables.skipKernelVersionCheck` option in the kube-proxy config, but this is not
recommended in production since it may cause problems with other nftables
users on the system.) ([kubernetes/kubernetes124152](https://github.com/kubernetes/kubernetes/pull/124152), [danwinship](https://github.com/danwinship)) [SIG Network]
- Use omitempty for optional Job Pod Failure Policy fields ([kubernetes/kubernetes126046](https://github.com/kubernetes/kubernetes/pull/126046), [mimowo](https://github.com/mimowo)) [SIG Apps]
- User can choose a different static policy option `SpreadPhysicalCPUsPreferredOption` to spread cpus across physical cpus for some specific applications ([kubernetes/kubernetes123733](https://github.com/kubernetes/kubernetes/pull/123733), [Jeffwan](https://github.com/Jeffwan)) [SIG Node]
- DRA: in the `pod.spec.recourceClaims` array, the `source` indirection is no longer necessary. Instead of e.g. `source: resourceClaimTemplateName: my-template`, one can write `resourceClaimTemplateName: my-template`. ([kubernetes/kubernetes125116](https://github.com/kubernetes/kubernetes/pull/125116), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- Fix code-generator client-gen to work with `api/v1`-like package structure. ([kubernetes/kubernetes125162](https://github.com/kubernetes/kubernetes/pull/125162), [sttts](https://github.com/sttts)) [SIG API Machinery and Apps]
- KEP-1880: Users of the new feature to add multiple service CIDR will use by default a dual-write strategy on the new ClusterIP allocators to avoid the problem of possible duplicate IPs allocated to Services when running skewed kube-apiservers using different allocators. They can opt-out of this behavior by enabled the feature gate DisableAllocatorDualWrite ([kubernetes/kubernetes122047](https://github.com/kubernetes/kubernetes/pull/122047), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Instrumentation and Testing]
- Kube-apiserver: ControllerRevision objects are now verified to contain valid JSON data in the `data` field. ([kubernetes/kubernetes125549](https://github.com/kubernetes/kubernetes/pull/125549), [liggitt](https://github.com/liggitt)) [SIG API Machinery and Apps]
- Update the feature MultiCIDRServiceAllocator to beta (disabled by default). Users need to enable the feature gate and the networking v1beta1 group to be able to use this new feature, that allows to dynamically reconfigure Service CIDR ranges. ([kubernetes/kubernetes125021](https://github.com/kubernetes/kubernetes/pull/125021), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, CLI, Etcd, Instrumentation, Network and Testing]
- When the featuregate AnonymousAuthConfigurableEndpoints is enabled users can update the AuthenticationConfig file with endpoints for with anonymous requests are alllowed. ([kubernetes/kubernetes124917](https://github.com/kubernetes/kubernetes/pull/124917), [vinayakankugoyal](https://github.com/vinayakankugoyal)) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
- Fixed incorrect "v1 Binding is deprecated in v1.6+" warning in kube-scheduler log. ([kubernetes/kubernetes125540](https://github.com/kubernetes/kubernetes/pull/125540), [pohly](https://github.com/pohly)) [SIG API Machinery]
- Added the feature gates `StrictCostEnforcementForVAP` and `StrictCostEnforcementForWebhooks` to enforce the strct cost calculation for CEL extended libraries. It is strongly recommended to turn on the feature gates as early as possible. ([kubernetes/kubernetes124675](https://github.com/kubernetes/kubernetes/pull/124675), [cici37](https://github.com/cici37)) [SIG API Machinery, Auth, Node and Testing]
- Component-base/logs: when compiled with Go >= 1.21, component-base will automatically configure the slog default logger together with initializing klog. ([kubernetes/kubernetes120696](https://github.com/kubernetes/kubernetes/pull/120696), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Storage and Testing]
- DRA: client-side validation of a ResourceHandle would have accepted a missing DriverName, whereas server-side validation then would have raised an error. ([kubernetes/kubernetes124075](https://github.com/kubernetes/kubernetes/pull/124075), [pohly](https://github.com/pohly)) [SIG Apps]
- Fix Deep Copy issue in getting controller reference ([kubernetes/kubernetes124116](https://github.com/kubernetes/kubernetes/pull/124116), [HiranmoyChowdhury](https://github.com/HiranmoyChowdhury)) [SIG API Machinery and Release]
- Fix the comment for the Job's managedBy field ([kubernetes/kubernetes124793](https://github.com/kubernetes/kubernetes/pull/124793), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. ([kubernetes/kubernetes124553](https://github.com/kubernetes/kubernetes/pull/124553), [pmalek](https://github.com/pmalek)) [SIG API Machinery]
- Graduate MatchLabelKeys/MismatchLabelKeys feature in PodAffinity/PodAntiAffinity to Beta ([kubernetes/kubernetes123638](https://github.com/kubernetes/kubernetes/pull/123638), [sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Scheduling and Testing]
- Graduated the `DisableNodeKubeProxyVersion` feature gate to beta. By default, the kubelet no longer attempts to set the `.status.kubeProxyVersion` field for its associated Node. ([kubernetes/kubernetes123845](https://github.com/kubernetes/kubernetes/pull/123845), [HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Cloud Provider, Network, Node and Testing]
- Improved scheduling performance when many nodes, and prefilter returns 1-2 nodes (e.g. daemonset)

For developers of out-of-tree PostFilter plugins, note that the semantics of NodeToStatusMap are changing: A node with an absent value in the NodeToStatusMap should be interpreted as having an UnschedulableAndUnresolvable status ([kubernetes/kubernetes125197](https://github.com/kubernetes/kubernetes/pull/125197), [gabesaba](https://github.com/gabesaba)) [SIG Scheduling]
- K8s.io/apimachinery/pkg/util/runtime: new calls support handling panics and errors in the context where they occur. `PanicHandlers` and `ErrorHandlers` now must accept a context parameter for that. Log output is structured instead of unstructured. ([kubernetes/kubernetes121970](https://github.com/kubernetes/kubernetes/pull/121970), [pohly](https://github.com/pohly)) [SIG API Machinery and Instrumentation]
- Kube-apiserver: the `--encryption-provider-config` file is now loaded with strict deserialization, which fails if the config file contains duplicate or unknown fields. This protects against accidentally running with config files that are malformed, mis-indented, or have typos in field names, and getting unexpected behavior. When `--encryption-provider-config-automatic-reload` is used, new encryption config files that contain typos after the kube-apiserver is running are treated as invalid and the last valid config is used. ([kubernetes/kubernetes124912](https://github.com/kubernetes/kubernetes/pull/124912), [enj](https://github.com/enj)) [SIG API Machinery and Auth]
- Kube-controller-manager removes deprecated command flags: --volume-host-cidr-denylist and --volume-host-allow-local-loopback ([kubernetes/kubernetes124017](https://github.com/kubernetes/kubernetes/pull/124017), [carlory](https://github.com/carlory)) [SIG API Machinery, Apps, Cloud Provider and Storage]
- Kube-controller-manager: the `horizontal-pod-autoscaler-upscale-delay` and `horizontal-pod-autoscaler-downscale-delay` flags have been removed (deprecated and non-functional since v1.12) ([kubernetes/kubernetes124948](https://github.com/kubernetes/kubernetes/pull/124948), [SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Apps and Autoscaling]
- Support fine-grained supplemental groups policy (KEP-3619), which enables fine-grained control for supplementary groups in the first container processes. You can choose whether to include groups defined in the container image(/etc/groups) for the container's primary uid or not. ([kubernetes/kubernetes117842](https://github.com/kubernetes/kubernetes/pull/117842), [everpeace](https://github.com/everpeace)) [SIG API Machinery, Apps and Node]
- The kube-proxy nodeportAddresses / --nodeport-addresses option now
accepts the value "primary", meaning to only listen for NodePort connections
on the node's primary IPv4 and/or IPv6 address (according to the Node object).
This is strongly recommended, if you were not previously using
--nodeport-addresses, to avoid surprising behavior.

(This behavior is enabled by default with the nftables backend; you would
need to explicitly request `--nodeport-addresses 0.0.0.0/0,::/0` there to get
the traditional "listen on all interfaces" behavior.) ([kubernetes/kubernetes123105](https://github.com/kubernetes/kubernetes/pull/123105), [danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Windows]

API Change
- Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. ([kubernetes/kubernetes124553](https://github.com/kubernetes/kubernetes/pull/124553), [pmalek](https://github.com/pmalek)) [SIG API Machinery]
- Fixes a 1.30.0 regression in openapi descriptions of imagePullSecrets and hostAliases fields to mark the fields used as keys in those lists as either defaulted or required. ([kubernetes/kubernetes124694](https://github.com/kubernetes/kubernetes/pull/124694), [pmalek](https://github.com/pmalek)) [SIG API Machinery]
- Added (alpha) support for the `managedBy` field on Jobs. Jobs with a custom value of this field - any value other than `kubernetes.io/job-controller` - were skipped by the job controller, and their reconciliation was delegated to an external controller, indicated by the value of the field. Jobs that didn't have this field at all, or where the field value was the reserved string `kubernetes.io/job-controller`, were reconciled by the built-in job controller.
([kubernetes/kubernetes123273](https://github.com/kubernetes/kubernetes/pull/123273), [mimowo](https://github.com/mimowo))
- Added alpha-level support for the SuccessPolicy in Jobs.
([kubernetes/kubernetes123412](https://github.com/kubernetes/kubernetes/pull/123412), [tenzen-y](https://github.com/tenzen-y))
- Added the `CEL` library for IP Addresses and CIDRs. This was made available for use starting from version `1.31`.
([kubernetes/kubernetes121912](https://github.com/kubernetes/kubernetes/pull/121912), [JoelSpeed](https://github.com/JoelSpeed))
- Allowed container runtimes to fix an image garbage collection bug by adding an `image_id` field to the CRI Container message.
([kubernetes/kubernetes123508](https://github.com/kubernetes/kubernetes/pull/123508), [saschagrunert](https://github.com/saschagrunert))
- Dynamic Resource Allocation: DRA drivers can now use "structured parameters" to let the scheduler handle claim allocation.
([kubernetes/kubernetes123516](https://github.com/kubernetes/kubernetes/pull/123516), [pohly](https://github.com/pohly))
- Fixed accidental enablement of the new alpha `optionalOldSelf` API field in `CustomResourceDefinition` validation rules, which should only have been allowed to be set when the `CRDValidationRatcheting` feature gate is enabled.
([kubernetes/kubernetes122329](https://github.com/kubernetes/kubernetes/pull/122329), [jpbetz](https://github.com/jpbetz))
- Implemented the `prescore` extension point for the `volumeBinding` plugin. It now returns skip if it doesn't do anything in Score.
([kubernetes/kubernetes115768](https://github.com/kubernetes/kubernetes/pull/115768), [AxeZhan](https://github.com/AxeZhan))
- Kubelet would fail if NodeSwap was used with LimitedSwap and cgroupv1 node.
([kubernetes/kubernetes123738](https://github.com/kubernetes/kubernetes/pull/123738), [kannon92](https://github.com/kannon92))
- Promoted `AdmissionWebhookMatchConditions` to GA. The feature is now stable, and the feature gate is now locked to default.
([kubernetes/kubernetes123560](https://github.com/kubernetes/kubernetes/pull/123560), [ivelichkovich](https://github.com/ivelichkovich))
- Structured Authentication Configuration now supports `DiscoveryURL`. If specified, `discoveryURL` overrides the URL used to fetch discovery information. This is for scenarios where the well-known and jwks endpoints are hosted at a different location than the issuer (such as locally in the cluster).
([kubernetes/kubernetes123527](https://github.com/kubernetes/kubernetes/pull/123527), [aramase](https://github.com/aramase))
- The `StorageVersionMigration` API, previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes.
([kubernetes/kubernetes123344](https://github.com/kubernetes/kubernetes/pull/123344), [nilekhc](https://github.com/nilekhc))
- When configuring a JWT authenticator:

If `username.expression` used 'claims.email', then 'claims.email_verified' must have been used in `username.expression` or `extra[*].valueExpression` or `claimValidationRules[*].expression`. An example claim validation rule expression that matches the validation automatically applied when `username.claim` is set to 'email' is 'claims.?email_verified.orValue(true)'.
([kubernetes/kubernetes123737](https://github.com/kubernetes/kubernetes/pull/123737), [enj](https://github.com/enj))
- `readOnly` volumes now support recursive read-only mounts for kernel versions >= 5.12."
([kubernetes/kubernetes123180](https://github.com/kubernetes/kubernetes/pull/123180), [AkihiroSuda](https://github.com/AkihiroSuda))
- cri-api: Implemented KEP-3857: Recursive Read-only (RRO) mounts.
([kubernetes/kubernetes123272](https://github.com/kubernetes/kubernetes/pull/123272), [AkihiroSuda](https://github.com/AkihiroSuda))
- kube-apiserver: the AuthenticationConfiguration type accepted in `--authentication-config` files has been promoted to `apiserver.config.k8s.io/v1beta1`.
([kubernetes/kubernetes123696](https://github.com/kubernetes/kubernetes/pull/123696), [aramase](https://github.com/aramase))
- kubelet allowed specifying a custom root directory for pod logs (instead of the default /var/log/pods) using the `podLogsDir` key in kubelet configuration.
([kubernetes/kubernetes112957](https://github.com/kubernetes/kubernetes/pull/112957), [mxpv](https://github.com/mxpv))
- resource.k8s.io/ResourceClaim (alpha API): The strategic merge patch strategy for the `status.reservedFor` array was changed so that a strategic-merge-patch can now add individual entries. This change may break clients using strategic merge patch to update status, which rely on the previous behavior (replacing the entire array).
([kubernetes/kubernetes122276](https://github.com/kubernetes/kubernetes/pull/122276), [pohly](https://github.com/pohly))
- Added a CBOR implementation of `runtime.Serializer`. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. ([kubernetes/kubernetes122881](https://github.com/kubernetes/kubernetes/pull/122881), [benluddy](https://github.com/benluddy))
- Added a alpha feature, behind the `RelaxedEnvironmentVariableValidation` feature gate.
When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
of environment variables for containers in Pods. ([kubernetes/kubernetes123385](https://github.com/kubernetes/kubernetes/pull/123385), [HirazawaUi](https://github.com/HirazawaUi))
- Added a new (alpha) field, `trafficDistribution`, to the Service `spec` to express preferences for traffic distribution to endpoints. Enabled through the `ServiceTrafficDistribution` feature gate. ([kubernetes/kubernetes123487](https://github.com/kubernetes/kubernetes/pull/123487), [gauravkghildiyal](https://github.com/gauravkghildiyal))
- Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.
The "audienceMatchPolicy" can be empty (or unset) when a single audience is specified in the "audiences" field.
The "audienceMatchPolicy" must be set to "MatchAny" when multiple audiences are specified in the "audiences" field. ([kubernetes/kubernetes123165](https://github.com/kubernetes/kubernetes/pull/123165), [aramase](https://github.com/aramase))
- Added consistent vanity import to files and provided tooling for verifying and updating them. ([kubernetes/kubernetes120642](https://github.com/kubernetes/kubernetes/pull/120642), [jcchavezs](https://github.com/jcchavezs))
- Added the `disable-force-detach` CLI option for `kube-controller-manager`. By default, it's set to `false`. When enabled, it prevents force detaching volumes based on maximum unmount time and node status. If activated, the non-graceful node shutdown feature must be used to recover from node failure. Additionally, if a pod needs to be forcibly terminated at the risk of corruption, the appropriate VolumeAttachment object must be deleted. ([kubernetes/kubernetes120344](https://github.com/kubernetes/kubernetes/pull/120344), [rohitssingh](https://github.com/rohitssingh))
- Added to `MutableFeatureGate` the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. ([kubernetes/kubernetes122647](https://github.com/kubernetes/kubernetes/pull/122647), [benluddy](https://github.com/benluddy))
- Aggregated discovery supports both `v2beta1` and v2 types and feature is promoted to GA. ([kubernetes/kubernetes122882](https://github.com/kubernetes/kubernetes/pull/122882), [Jefftree](https://github.com/Jefftree))
- Alpha support for field selectors on custom resources has been added. With the `CustomResourceFieldSelectors` feature gate enabled, the CustomResourceDefinition API now allows specifying `selectableFields`. Listing a field there enables filtering custom resources for that CustomResourceDefinition in list or watch requests. ([kubernetes/kubernetes122717](https://github.com/kubernetes/kubernetes/pull/122717), [jpbetz](https://github.com/jpbetz))
- AppArmor profiles can now be configured through fields on the `PodSecurityContext` and container `SecurityContext`. The beta AppArmor annotations are deprecated, and AppArmor status is no longer included in the node ready condition. ([kubernetes/kubernetes123435](https://github.com/kubernetes/kubernetes/pull/123435), [tallclair](https://github.com/tallclair))
- Contextual logging is now in beta and enabled by default. Check out the [KEP](https://github.com/kubernetes/enhancements/issues/3077) and [official documentation](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) for more details. ([kubernetes/kubernetes122589](https://github.com/kubernetes/kubernetes/pull/122589), [pohly](https://github.com/pohly))
- Enabled concurrent log rotation in kubelet. You can now configure the maximum number of concurrent rotations with the `containerLogMaxWorkers` setting, and adjust the monitoring interval with `containerLogMonitorInterval`. ([kubernetes/kubernetes114301](https://github.com/kubernetes/kubernetes/pull/114301), [harshanarayana](https://github.com/harshanarayana))
- Graduated pod scheduling gates to general availability.
The `PodSchedulingReadiness` feature gate no longer has any effect, and the
`.spec.schedulingGates` field is always available within the Pod and PodTemplate APIs. ([kubernetes/kubernetes123575](https://github.com/kubernetes/kubernetes/pull/123575), [Huang-Wei](https://github.com/Huang-Wei))
- Graduated support for `minDomains` in pod topology spread constraints, to general availability.
The `MinDomainsInPodTopologySpread` feature gate no longer has any effect, and the field is
always available within the Pod and PodTemplate APIs. ([kubernetes/kubernetes123481](https://github.com/kubernetes/kubernetes/pull/123481), [sanposhiho](https://github.com/sanposhiho))
- In kubelet configuration, the `.memorySwap.swapBehavior` field now accepts a new value `NoSwap`, which becomes the default if unspecified. The previously accepted `UnlimitedSwap` value has been dropped.
([kubernetes/kubernetes122745](https://github.com/kubernetes/kubernetes/pull/122745), [kannon92](https://github.com/kannon92))
- Kube-apiserver: the AuthorizationConfiguration type accepted in `--authorization-config` files has been promoted to `apiserver.config.k8s.io/v1beta1`. ([kubernetes/kubernetes123640](https://github.com/kubernetes/kubernetes/pull/123640), [liggitt](https://github.com/liggitt))
- OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. ([kubernetes/kubernetes123568](https://github.com/kubernetes/kubernetes/pull/123568), [enj](https://github.com/enj))
- Removed note that `hostAliases` are not supported on hostNetwork Pods from the PodSpec API. The feature has been supported since v1.8. ([kubernetes/kubernetes122422](https://github.com/kubernetes/kubernetes/pull/122422), [neolit123](https://github.com/neolit123))
- Structured Authentication Configuration now supports configuring multiple JWT authenticators. The maximum allowed JWT authenticators in the authentication configuration is 64. ([kubernetes/kubernetes123431](https://github.com/kubernetes/kubernetes/pull/123431), [aramase](https://github.com/aramase))
- Text logging in Kubernetes components now uses [textlogger](https://pkg.go.dev/k8s.io/klog/v2v2.120.0/textlogger). The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. ([kubernetes/kubernetes#114672](https://github.com/kubernetes/kubernetes/pull/114672), [pohly](https://github.com/pohly))
- The API server now detects and fails on startup if there are conflicting issuers between JWT authenticators and service account configurations. Previously, such configurations would run but could be inconsistently effective depending on the credential. ([kubernetes/kubernetes123561](https://github.com/kubernetes/kubernetes/pull/123561), [enj](https://github.com/enj))
- The JWT authenticator configuration set via the `--authentication-config` flag is now dynamically reloaded as the file changes on disk. ([kubernetes/kubernetes123525](https://github.com/kubernetes/kubernetes/pull/123525), [enj](https://github.com/enj))
- The `StructuredAuthenticationConfiguration` feature is now beta and enabled. ([kubernetes/kubernetes123719](https://github.com/kubernetes/kubernetes/pull/123719), [enj](https://github.com/enj))
- The `kube_codegen` tool now ignores the vendor folder during code generation.
([kubernetes/kubernetes122729](https://github.com/kubernetes/kubernetes/pull/122729), [jparrill](https://github.com/jparrill))
- The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. ([kubernetes/kubernetes123529](https://github.com/kubernetes/kubernetes/pull/123529), [thockin](https://github.com/thockin))
- Updated an audit annotation key used by the `…/serviceaccounts/<name>/token` resource handler.
The annotation used to persist the issued credential identifier is now `authentication.kubernetes.io/issued-credential-id`. ([kubernetes/kubernetes123098](https://github.com/kubernetes/kubernetes/pull/123098), [munnerz](https://github.com/munnerz)) [SIG Auth]
- Users are now allowed to mutate `FSGroupPolicy` and `PodInfoOnMount` in `CSIDriver.Spec`. ([kubernetes/kubernetes116209](https://github.com/kubernetes/kubernetes/pull/116209), [haoruan](https://github.com/haoruan))
- ValidatingAdmissionPolicy was promoted to GA and will be `enabled` by default. ([kubernetes/kubernetes123405](https://github.com/kubernetes/kubernetes/pull/123405), [cici37](https://github.com/cici37))
- When scheduling a mix of pods using `ResourceClaims` and others that don't, scheduling a pod with `ResourceClaims` has a lower impact on scheduling latency. ([kubernetes/kubernetes121876](https://github.com/kubernetes/kubernetes/pull/121876), [pohly](https://github.com/pohly))
- When working with client-go events, it's now recommended to use `NewEventBroadcasterAdapterWithContext` instead of `NewEventBroadcasterAdapter` if contextual logging support is needed. ([kubernetes/kubernetes122142](https://github.com/kubernetes/kubernetes/pull/122142), [pohly](https://github.com/pohly))
- A new (alpha) field, `trafficDistribution`, has been added to the Service `spec`.
This field provides a way to express preferences for how traffic is distributed to the endpoints for a Service.
It can be enabled through the `ServiceTrafficDistribution` feature gate. ([kubernetes/kubernetes123487](https://github.com/kubernetes/kubernetes/pull/123487), [gauravkghildiyal](https://github.com/gauravkghildiyal)) [SIG API Machinery, Apps and Network]
- Add alpha-level support for the SuccessPolicy in Jobs ([kubernetes/kubernetes123412](https://github.com/kubernetes/kubernetes/pull/123412), [tenzen-y](https://github.com/tenzen-y)) [SIG API Machinery, Apps and Testing]
- Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any
value other than `kubernetes.io/job-controller` - are skipped by the job controller, and their
reconciliation is delegated to an external controller, indicated by the value of the field. Jobs that
don't have this field at all, or where the field value is the reserved string `kubernetes.io/job-controller`,
are reconciled by the built-in job controller. ([kubernetes/kubernetes123273](https://github.com/kubernetes/kubernetes/pull/123273), [mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps and Testing]
- Added a alpha feature, behind the `RelaxedEnvironmentVariableValidation` feature gate.
When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
of environment variables for containers in Pods. ([kubernetes/kubernetes123385](https://github.com/kubernetes/kubernetes/pull/123385), [HirazawaUi](https://github.com/HirazawaUi)) [SIG Apps, Node and Testing]
- Added alpha support for field selectors on custom resources.
Provided that the `CustomResourceFieldSelectors` feature gate is enabled, the CustomResourceDefinition
API now lets you specify `selectableFields`. Listing a field there allows filtering custom resources for that
CustomResourceDefinition in **list** or **watch** requests. ([kubernetes/kubernetes122717](https://github.com/kubernetes/kubernetes/pull/122717), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery]
- Added support for configuring multiple JWT authenticators in Structured Authentication Configuration. The maximum allowed JWT authenticators in the authentication configuration is 64. ([kubernetes/kubernetes123431](https://github.com/kubernetes/kubernetes/pull/123431), [aramase](https://github.com/aramase)) [SIG Auth and Testing]
- Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA ([kubernetes/kubernetes122882](https://github.com/kubernetes/kubernetes/pull/122882), [Jefftree](https://github.com/Jefftree)) [SIG API Machinery and Testing]
- Allowing container runtimes to fix an image garbage collection bug by adding an `image_id` field to the CRI Container message. ([kubernetes/kubernetes123508](https://github.com/kubernetes/kubernetes/pull/123508), [saschagrunert](https://github.com/saschagrunert)) [SIG Node]
- AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext.
- The beta AppArmor annotations are deprecated.
- AppArmor status is no longer included in the node ready condition ([kubernetes/kubernetes123435](https://github.com/kubernetes/kubernetes/pull/123435), [tallclair](https://github.com/tallclair)) [SIG API Machinery, Apps, Auth, Node and Testing]
- Conflicting issuers between JWT authenticators and service account config are now detected and fail on API server startup. Previously such a config would run but would be inconsistently effective depending on the credential. ([kubernetes/kubernetes123561](https://github.com/kubernetes/kubernetes/pull/123561), [enj](https://github.com/enj)) [SIG API Machinery and Auth]
- Dynamic Resource Allocation: DRA drivers may now use "structured parameters" to let the scheduler handle claim allocation. ([kubernetes/kubernetes123516](https://github.com/kubernetes/kubernetes/pull/123516), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]
- Graduated pod scheduling gates to general availability.
The `PodSchedulingReadiness` feature gate no longer has any effect, and the
`.spec.schedulingGates` field is always available within the Pod and PodTemplate APIs. ([kubernetes/kubernetes123575](https://github.com/kubernetes/kubernetes/pull/123575), [Huang-Wei](https://github.com/Huang-Wei)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Graduated support for `minDomains` in pod topology spread constraints, to general availability.
The `MinDomainsInPodTopologySpread` feature gate no longer has any effect, and the field is
always available within the Pod and PodTemplate APIs. ([kubernetes/kubernetes123481](https://github.com/kubernetes/kubernetes/pull/123481), [sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Scheduling and Testing]
- JWT authenticator config set via the --authentication-config flag is now dynamically reloaded as the file changes on disk. ([kubernetes/kubernetes123525](https://github.com/kubernetes/kubernetes/pull/123525), [enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
- Kube-apiserver: the AuthenticationConfiguration type accepted in `--authentication-config` files has been promoted to `apiserver.config.k8s.io/v1beta1`. ([kubernetes/kubernetes123696](https://github.com/kubernetes/kubernetes/pull/123696), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Kube-apiserver: the AuthorizationConfiguration type accepted in `--authorization-config` files has been promoted to `apiserver.config.k8s.io/v1beta1`. ([kubernetes/kubernetes123640](https://github.com/kubernetes/kubernetes/pull/123640), [liggitt](https://github.com/liggitt)) [SIG Auth and Testing]
- Kubelet should fail if NodeSwap is used with LimitedSwap and cgroupv1 node. ([kubernetes/kubernetes123738](https://github.com/kubernetes/kubernetes/pull/123738), [kannon92](https://github.com/kannon92)) [SIG API Machinery, Node and Testing]
- Kubelet: a custom root directory for pod logs (instead of default /var/log/pods) can be specified using the `podLogsDir`
key in kubelet configuration. ([kubernetes/kubernetes112957](https://github.com/kubernetes/kubernetes/pull/112957), [mxpv](https://github.com/mxpv)) [SIG API Machinery, Node, Scalability and Testing]
- Kubelet: the `.memorySwap.swapBehavior` field in kubelet configuration accepts a new value `NoSwap` and makes this the default if unspecified; the previously accepted `UnlimitedSwap` value has been dropped. ([kubernetes/kubernetes122745](https://github.com/kubernetes/kubernetes/pull/122745), [kannon92](https://github.com/kannon92)) [SIG API Machinery, Node and Testing]
- OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. ([kubernetes/kubernetes123568](https://github.com/kubernetes/kubernetes/pull/123568), [enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
- PodSpec API: remove note that hostAliases are not supported on hostNetwork Pods. The feature has been supported since v1.8. ([kubernetes/kubernetes122422](https://github.com/kubernetes/kubernetes/pull/122422), [neolit123](https://github.com/neolit123)) [SIG API Machinery and Apps]
- Promote AdmissionWebhookMatchConditions to GA. The feature is now stable and the feature gate is now locked to default. ([kubernetes/kubernetes123560](https://github.com/kubernetes/kubernetes/pull/123560), [ivelichkovich](https://github.com/ivelichkovich)) [SIG API Machinery and Testing]
- Structured Authentication Configuration now supports `DiscoveryURL`.
discoveryURL if specified, overrides the URL used to fetch discovery information.
This is for scenarios where the well-known and jwks endpoints are hosted at a different
location than the issuer (such as locally in the cluster). ([kubernetes/kubernetes123527](https://github.com/kubernetes/kubernetes/pull/123527), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Support Recursive Read-only (RRO) mounts (KEP-3857) ([kubernetes/kubernetes123180](https://github.com/kubernetes/kubernetes/pull/123180), [AkihiroSuda](https://github.com/AkihiroSuda)) [SIG API Machinery, Apps, Node and Testing]
- The StructuredAuthenticationConfiguration feature is now beta and enabled by default. ([kubernetes/kubernetes123719](https://github.com/kubernetes/kubernetes/pull/123719), [enj](https://github.com/enj)) [SIG API Machinery and Auth]
- The `StorageVersionMigration` API, which was previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes. ([kubernetes/kubernetes123344](https://github.com/kubernetes/kubernetes/pull/123344), [nilekhc](https://github.com/nilekhc)) [SIG API Machinery, Apps, Auth, CLI and Testing]
- The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. ([kubernetes/kubernetes123529](https://github.com/kubernetes/kubernetes/pull/123529), [thockin](https://github.com/thockin)) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
- ValidatingAdmissionPolicy is promoted to GA and will be enabled by default. ([kubernetes/kubernetes123405](https://github.com/kubernetes/kubernetes/pull/123405), [cici37](https://github.com/cici37)) [SIG API Machinery, Apps, Auth and Testing]
- When configuring a JWT authenticator:

If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
username.expression or extra[*].valueExpression or claimValidationRules[*].expression.
An example claim validation rule expression that matches the validation automatically
applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true)'. ([kubernetes/kubernetes123737](https://github.com/kubernetes/kubernetes/pull/123737), [enj](https://github.com/enj)) [SIG API Machinery and Auth]
- Added a CBOR implementation of `runtime.Serializer`. Until CBOR graduates to Alpha, API servers will refuse to start if configured with CBOR support. ([kubernetes/kubernetes122881](https://github.com/kubernetes/kubernetes/pull/122881), [benluddy](https://github.com/benluddy)) [SIG API Machinery]
- Added audienceMatchPolicy field to AuthenticationConfiguration and support for configuring multiple audiences.

- The "audienceMatchPolicy" can be empty (or unset) when a single audience is specified in the "audiences" field.
- The "audienceMatchPolicy" must be set to "MatchAny" when multiple audiences are specified in the "audiences" field. ([kubernetes/kubernetes123165](https://github.com/kubernetes/kubernetes/pull/123165), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Contextual logging is now beta and enabled by default. ([kubernetes/kubernetes122589](https://github.com/kubernetes/kubernetes/pull/122589), [pohly](https://github.com/pohly)) [SIG Instrumentation]
- Cri-api: KEP-3857: Recursive Read-only (RRO) mounts ([kubernetes/kubernetes123272](https://github.com/kubernetes/kubernetes/pull/123272), [AkihiroSuda](https://github.com/AkihiroSuda)) [SIG Node]
- Enabled a mechanism for concurrent log rotatation via `kubelet` using a configuration entity of `containerLogMaxWorkers` which controls the maximum number of concurrent rotation that can be performed and an interval configuration of `containerLogMonitorInterval` that can aid is configuring the monitoring duration to best suite your cluster's log generation standards. ([kubernetes/kubernetes114301](https://github.com/kubernetes/kubernetes/pull/114301), [harshanarayana](https://github.com/harshanarayana)) [SIG API Machinery, Node and Testing]
- Text logging in Kubernetes components now uses [textlogger](https://pkg.go.dev/k8s.io/klog/v2v2.120.0/textlogger). The same split streams of info and error log entries with buffering of info entries is now also supported for text output (off by default, alpha feature). Previously, this was only supported for JSON. Performance is better also without split streams. ([kubernetes/kubernetes#114672](https://github.com/kubernetes/kubernetes/pull/114672), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
- This change adds the following CLI option for `kube-controller-manager`:
- `disable-force-detach` (defaults to `false`): Prevent force detaching volumes based on maximum unmount time and node status. If enabled, the non-graceful node shutdown feature must be used to recover from node failure (see https://kubernetes.io/blog/2023/08/16/kubernetes-1-28-non-graceful-node-shutdown-ga/). If enabled and a pod must be forcibly terminated at the risk of corruption, then the appropriate VolumeAttachment object (see here: https://kubernetes.io/docs/reference/kubernetes-api/config-and-storage-resources/volume-attachment-v1/) must be deleted. ([kubernetes/kubernetes#120344](https://github.com/kubernetes/kubernetes/pull/120344), [rohitssingh](https://github.com/rohitssingh)) [SIG API Machinery, Apps, Storage and Testing]
- Updated an audit annotation key used by the `…/serviceaccounts/<name>/token` resource handler.
The annotation used to persist the issued credential identifier is now `authentication.kubernetes.io/issued-credential-id`. ([kubernetes/kubernetes123098](https://github.com/kubernetes/kubernetes/pull/123098), [munnerz](https://github.com/munnerz)) [SIG Auth]
- Add CEL library for IP Addresses and CIDRs. This will not be available for use until 1.31. ([kubernetes/kubernetes121912](https://github.com/kubernetes/kubernetes/pull/121912), [JoelSpeed](https://github.com/JoelSpeed)) [SIG API Machinery]
- Added to MutableFeatureGate the ability to override the default setting of feature gates, to allow default-enabling a feature on a component-by-component basis instead of for all affected components simultaneously. ([kubernetes/kubernetes122647](https://github.com/kubernetes/kubernetes/pull/122647), [benluddy](https://github.com/benluddy)) [SIG API Machinery and Cluster Lifecycle]
- Adds a rule on the kube_codegen tool to ignore vendor folder during the code generation. ([kubernetes/kubernetes122729](https://github.com/kubernetes/kubernetes/pull/122729), [jparrill](https://github.com/jparrill)) [SIG API Machinery and Cluster Lifecycle]
- Allow users to mutate FSGroupPolicy and PodInfoOnMount in CSIDriver.Spec ([kubernetes/kubernetes116209](https://github.com/kubernetes/kubernetes/pull/116209), [haoruan](https://github.com/haoruan)) [SIG API Machinery, Storage and Testing]
- Client-go events: `NewEventBroadcasterAdapterWithContext` should be used instead of `NewEventBroadcasterAdapter` if the goal is to support contextual logging. ([kubernetes/kubernetes122142](https://github.com/kubernetes/kubernetes/pull/122142), [pohly](https://github.com/pohly)) [SIG API Machinery, Instrumentation and Scheduling]
- Fixes accidental enablement of the new alpha `optionalOldSelf` API field in CustomResourceDefinition validation rules, which should only be allowed to be set when the CRDValidationRatcheting feature gate is enabled. ([kubernetes/kubernetes122329](https://github.com/kubernetes/kubernetes/pull/122329), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery]
- Implement `prescore` extension point for `volumeBinding` plugin. Return skip if it doesn't do anything in Score. ([kubernetes/kubernetes115768](https://github.com/kubernetes/kubernetes/pull/115768), [AxeZhan](https://github.com/AxeZhan)) [SIG Scheduling, Storage and Testing]
- Resource.k8s.io/ResourceClaim (alpha API): the strategic merge patch strategy for the `status.reservedFor` array was changed such that a strategic-merge-patch can add individual entries. This breaks clients using strategic merge patch to update status which rely on the previous behavior (replacing the entire array). ([kubernetes/kubernetes122276](https://github.com/kubernetes/kubernetes/pull/122276), [pohly](https://github.com/pohly)) [SIG API Machinery]
- When scheduling a mixture of pods using ResourceClaims and others which don't, scheduling a pod with ResourceClaims impacts scheduling latency less. ([kubernetes/kubernetes121876](https://github.com/kubernetes/kubernetes/pull/121876), [pohly](https://github.com/pohly)) [SIG API Machinery, Node, Scheduling and Testing]

API Change
- '`kube-apiserver`: adds `--authentication-config` flag for reading `AuthenticationConfiguration`
files. `--authentication-config` flag is mutually exclusive with the existing `--oidc-*`
flags.' ([kubernetes/kubernetes119142](https://github.com/kubernetes/kubernetes/pull/119142), [aramase](https://github.com/aramase))
- '`kube-scheduler` component config (`KubeSchedulerConfiguration`) `kubescheduler.config.k8s.io/v1beta3`
is removed in `v1.29`. Migrated `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`.' ([kubernetes/kubernetes119994](https://github.com/kubernetes/kubernetes/pull/119994), [SataQiu](https://github.com/SataQiu))
- A new sleep action for the `PreStop` lifecycle hook was added, allowing containers to pause for a specified duration before termination. ([kubernetes/kubernetes119026](https://github.com/kubernetes/kubernetes/pull/119026), [AxeZhan](https://github.com/AxeZhan))
- Added CEL expressions to `v1alpha1 AuthenticationConfiguration`. ([kubernetes/kubernetes121078](https://github.com/kubernetes/kubernetes/pull/121078), [aramase](https://github.com/aramase))
- Added Windows support for InPlace Pod Vertical Scaling feature. ([kubernetes/kubernetes112599](https://github.com/kubernetes/kubernetes/pull/112599), [fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added `ImageMaximumGCAge` field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([kubernetes/kubernetes121275](https://github.com/kubernetes/kubernetes/pull/121275), [haircommander](https://github.com/haircommander))
- Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([kubernetes/kubernetes118760](https://github.com/kubernetes/kubernetes/pull/118760), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release]
- Added `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints. ([kubernetes/kubernetes121034](https://github.com/kubernetes/kubernetes/pull/121034), [alexzielenski](https://github.com/alexzielenski))
- Added a new `ServiceCIDR` type that allows to dynamically configure the cluster range used to allocate `Service ClusterIPs` addresses. ([kubernetes/kubernetes116516](https://github.com/kubernetes/kubernetes/pull/116516), [aojea](https://github.com/aojea))
- Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`.
The new field is behind the `LoadBalancerIPMode` feature gate. ([kubernetes/kubernetes119937](https://github.com/kubernetes/kubernetes/pull/119937), [RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
- Added options for configuring `nf_conntrack_udp_timeout`, and `nf_conntrack_udp_timeout_stream` variables of netfilter conntrack subsystem. ([kubernetes/kubernetes120808](https://github.com/kubernetes/kubernetes/pull/120808), [aroradaman](https://github.com/aroradaman))
- Added support for CEL expressions to `v1alpha1 AuthorizationConfiguration` webhook `matchConditions`. ([kubernetes/kubernetes121223](https://github.com/kubernetes/kubernetes/pull/121223), [ritazh](https://github.com/ritazh))
- Added support for projecting `certificates.k8s.io/v1alpha1` ClusterTrustBundle objects into pods. ([kubernetes/kubernetes113374](https://github.com/kubernetes/kubernetes/pull/113374), [ahmedtd](https://github.com/ahmedtd))
- Added the `DisableNodeKubeProxyVersion` feature gate. If `DisableNodeKubeProxyVersion` is enabled, the `kubeProxyVersion` field is not set. ([kubernetes/kubernetes120954](https://github.com/kubernetes/kubernetes/pull/120954), [HirazawaUi](https://github.com/HirazawaUi))
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119800](https://github.com/kubernetes/kubernetes/pull/119800), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Fixed the API comments for the Job `Ready` field in status. ([kubernetes/kubernetes121765](https://github.com/kubernetes/kubernetes/pull/121765), [mimowo](https://github.com/mimowo))
- Fixed the API comments for the `FailIndex` Job pod failure policy action. ([kubernetes/kubernetes121764](https://github.com/kubernetes/kubernetes/pull/121764), [mimowo](https://github.com/mimowo))
- Go API: the `ResourceRequirements` struct was replaced with `VolumeResourceRequirements` for use with volumes. ([kubernetes/kubernetes118653](https://github.com/kubernetes/kubernetes/pull/118653), [pohly](https://github.com/pohly))
- Graduated `Job BackoffLimitPerIndex` feature to `beta`. ([kubernetes/kubernetes121356](https://github.com/kubernetes/kubernetes/pull/121356), [mimowo](https://github.com/mimowo))
- Marked the `onPodConditions` field as optional in `Job`'s pod failure policy. ([kubernetes/kubernetes120204](https://github.com/kubernetes/kubernetes/pull/120204), [mimowo](https://github.com/mimowo))
- Promoted `PodReadyToStartContainers` condition to `beta`. ([kubernetes/kubernetes119659](https://github.com/kubernetes/kubernetes/pull/119659), [kannon92](https://github.com/kannon92))
- The `flowcontrol.apiserver.k8s.io/v1beta3` `FlowSchema` and `PriorityLevelConfiguration` APIs has been promoted to `flowcontrol.apiserver.k8s.io/v1`, with the following changes:
- `PriorityLevelConfiguration`: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with `v1.28` API servers. In `v1.30`, explicit `0` values will be allowed in this field in the `v1` API.
The `flowcontrol.apiserver.k8s.io/v1beta3` APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to `v1.32`. ([kubernetes/kubernetes121089](https://github.com/kubernetes/kubernetes/pull/121089), [tkashem](https://github.com/tkashem))
- The `kube-proxy` command-line documentation was updated to clarify that
`--bind-address` does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. ([kubernetes/kubernetes120274](https://github.com/kubernetes/kubernetes/pull/120274), [danwinship](https://github.com/danwinship))
- The `kube-scheduler` `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([kubernetes/kubernetes117720](https://github.com/kubernetes/kubernetes/pull/117720), [kerthcet](https://github.com/kerthcet))
- The `matchLabelKeys/mismatchLabelKeys` feature is introduced to the hard/soft `PodAffinity/PodAntiAffinity`. ([kubernetes/kubernetes116065](https://github.com/kubernetes/kubernetes/pull/116065), [sanposhiho](https://github.com/sanposhiho))
- When updating a CRD, per-expression cost limit check are now skipped for `x-kubernetes-validations` rules of versions that are not mutated. ([kubernetes/kubernetes121460](https://github.com/kubernetes/kubernetes/pull/121460), [jiahuif](https://github.com/jiahuif))
- `CSINodeExpandSecret` feature has been promoted to `GA` in this release and is enabled
by default. The CSI drivers can make use of the `secretRef` values passed in `NodeExpansion`
request optionally sent by the CSI Client from this release onwards. ([kubernetes/kubernetes121303](https://github.com/kubernetes/kubernetes/pull/121303), [humblec](https://github.com/humblec))
- `NodeStageVolume` calls will now be retried if the CSI node driver is not running. ([kubernetes/kubernetes120330](https://github.com/kubernetes/kubernetes/pull/120330), [rohitssingh](https://github.com/rohitssingh))
- `PersistentVolumeLastPhaseTransitionTime` is now beta and enabled by default. ([kubernetes/kubernetes120627](https://github.com/kubernetes/kubernetes/pull/120627), [RomanBednar](https://github.com/RomanBednar))
- `ValidatingAdmissionPolicy` type checking now supports CRDs and API extensions types. ([kubernetes/kubernetes119109](https://github.com/kubernetes/kubernetes/pull/119109), [jiahuif](https://github.com/jiahuif))
- `kube-apiserver`: added `--authorization-config` flag for reading a configuration file containing an `apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration` object. The `--authorization-config` flag is mutually exclusive with `--authorization-modes` and `--authorization-webhook-*` flags. The `alpha` `StructuredAuthorizationConfiguration` feature flag must be enabled for `--authorization-config` to be specified. ([kubernetes/kubernetes120154](https://github.com/kubernetes/kubernetes/pull/120154), [palnabarun](https://github.com/palnabarun))
- `kube-proxy` now has a new nftables-based mode, available by running

`kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables`

This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)

At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the `--iptables-xxx` options.)

As this code is still very new, it has not been heavily optimized yet;
while it is expected to _eventually_ have better performance than the
iptables backend, very little performance testing has been done so far. ([kubernetes/kubernetes121046](https://github.com/kubernetes/kubernetes/pull/121046), [danwinship](https://github.com/danwinship))
- `kube-proxy`: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, `kube-proxy` will not install the `DROP` rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([kubernetes/kubernetes120354](https://github.com/kubernetes/kubernetes/pull/120354), [aroradaman](https://github.com/aroradaman))
- Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. ([kubernetes/kubernetes113374](https://github.com/kubernetes/kubernetes/pull/113374), [ahmedtd](https://github.com/ahmedtd)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Adds `optionalOldSelf` to `x-kubernetes-validations` to support ratcheting CRD schema constraints ([kubernetes/kubernetes121034](https://github.com/kubernetes/kubernetes/pull/121034), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery]
- Fix API comment for the Job Ready field in status ([kubernetes/kubernetes121765](https://github.com/kubernetes/kubernetes/pull/121765), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Fix API comments for the FailIndex Job pod failure policy action. ([kubernetes/kubernetes121764](https://github.com/kubernetes/kubernetes/pull/121764), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. ([kubernetes/kubernetes119026](https://github.com/kubernetes/kubernetes/pull/119026), [AxeZhan](https://github.com/AxeZhan)) [SIG API Machinery, Apps, Node and Testing]
- Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. ([kubernetes/kubernetes121275](https://github.com/kubernetes/kubernetes/pull/121275), [haircommander](https://github.com/haircommander)) [SIG API Machinery and Node]
- Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses ([kubernetes/kubernetes116516](https://github.com/kubernetes/kubernetes/pull/116516), [aojea](https://github.com/aojea)) [SIG API Machinery, Apps, Auth, CLI, Network and Testing]
- Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. ([kubernetes/kubernetes120954](https://github.com/kubernetes/kubernetes/pull/120954), [HirazawaUi](https://github.com/HirazawaUi)) [SIG API Machinery, Apps and Node]
- Added Windows support for InPlace Pod Vertical Scaling feature. ([kubernetes/kubernetes112599](https://github.com/kubernetes/kubernetes/pull/112599), [fabi200123](https://github.com/fabi200123)) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added `UserNamespacesPodSecurityStandards` feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting: `spec[.*].securityContext.[runAsNonRoot,runAsUser]`.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. ([kubernetes/kubernetes118760](https://github.com/kubernetes/kubernetes/pull/118760), [saschagrunert](https://github.com/saschagrunert)) [SIG API Machinery, Auth, Node and Release]
- Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. ([kubernetes/kubernetes120808](https://github.com/kubernetes/kubernetes/pull/120808), [aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network]
- Adds CEL expressions to v1alpha1 AuthenticationConfiguration. ([kubernetes/kubernetes121078](https://github.com/kubernetes/kubernetes/pull/121078), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. ([kubernetes/kubernetes121223](https://github.com/kubernetes/kubernetes/pull/121223), [ritazh](https://github.com/ritazh)) [SIG API Machinery and Auth]
- CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the `secretRef` values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. ([kubernetes/kubernetes121303](https://github.com/kubernetes/kubernetes/pull/121303), [humblec](https://github.com/humblec)) [SIG API Machinery, Apps and Storage]
- Graduate Job BackoffLimitPerIndex feature to Beta ([kubernetes/kubernetes121356](https://github.com/kubernetes/kubernetes/pull/121356), [mimowo](https://github.com/mimowo)) [SIG Apps]
- Kube-apiserver: adds --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. ([kubernetes/kubernetes120154](https://github.com/kubernetes/kubernetes/pull/120154), [palnabarun](https://github.com/palnabarun)) [SIG API Machinery, Auth and Testing]
- Kube-proxy now has a new nftables-based mode, available by running

kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables

This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)

At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the `--iptables-xxx` options.)

As this code is still very new, it has not been heavily optimized yet;
while it is expected to _eventually_ have better performance than the
iptables backend, very little performance testing has been done so far. ([kubernetes/kubernetes121046](https://github.com/kubernetes/kubernetes/pull/121046), [danwinship](https://github.com/danwinship)) [SIG API Machinery and Network]
- Kube-proxy: Added an option/flag for configuring the `nf_conntrack_tcp_be_liberal` sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. ([kubernetes/kubernetes120354](https://github.com/kubernetes/kubernetes/pull/120354), [aroradaman](https://github.com/aroradaman)) [SIG API Machinery and Network]
- PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. ([kubernetes/kubernetes120627](https://github.com/kubernetes/kubernetes/pull/120627), [RomanBednar](https://github.com/RomanBednar)) [SIG Storage]
- Promote PodReadyToStartContainers condition to beta. ([kubernetes/kubernetes119659](https://github.com/kubernetes/kubernetes/pull/119659), [kannon92](https://github.com/kannon92)) [SIG Node and Testing]
- The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:
- PriorityLevelConfiguration: the `.spec.limited.nominalConcurrencyShares` field defaults to `30` only if the field is omitted (v1beta3 also defaulted an explicit `0` value to `30`). Specifying an explicit `0` value is not allowed in the `v1` version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit `0` values will be allowed in this field in the `v1` API.
The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the `v1` APIs. Transition clients and manifests to use the `v1` APIs before upgrading to v1.32. ([kubernetes/kubernetes121089](https://github.com/kubernetes/kubernetes/pull/121089), [tkashem](https://github.com/tkashem)) [SIG API Machinery and Testing]
- The kube-proxy command-line documentation was updated to clarify that
`--bind-address` does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. ([kubernetes/kubernetes120274](https://github.com/kubernetes/kubernetes/pull/120274), [danwinship](https://github.com/danwinship)) [SIG Network]
- The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. ([kubernetes/kubernetes116065](https://github.com/kubernetes/kubernetes/pull/116065), [sanposhiho](https://github.com/sanposhiho)) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing]
- ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. ([kubernetes/kubernetes119109](https://github.com/kubernetes/kubernetes/pull/119109), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery, Apps, Auth and Testing]
- When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. ([kubernetes/kubernetes121460](https://github.com/kubernetes/kubernetes/pull/121460), [jiahuif](https://github.com/jiahuif)) [SIG API Machinery]
- Added a new `ipMode` field to the `.status` of Services where `type` is set to `LoadBalancer`.
The new field is behind the `LoadBalancerIPMode` feature gate. ([kubernetes/kubernetes119937](https://github.com/kubernetes/kubernetes/pull/119937), [RyanAoh](https://github.com/RyanAoh)) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119800](https://github.com/kubernetes/kubernetes/pull/119800), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Go API: the ResourceRequirements struct needs to be replaced with VolumeResourceRequirements for use with volumes. ([kubernetes/kubernetes118653](https://github.com/kubernetes/kubernetes/pull/118653), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling, Storage and Testing]
- Kube-apiserver: adds --authentication-config flag for reading AuthenticationConfiguration files. --authentication-config flag is mutually exclusive with the existing --oidc-* flags. ([kubernetes/kubernetes119142](https://github.com/kubernetes/kubernetes/pull/119142), [aramase](https://github.com/aramase)) [SIG API Machinery, Auth and Testing]
- Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta3 is removed in v1.29. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([kubernetes/kubernetes119994](https://github.com/kubernetes/kubernetes/pull/119994), [SataQiu](https://github.com/SataQiu)) [SIG Scheduling and Testing]
- Mark the onPodConditions field as optional in Job's pod failure policy. ([kubernetes/kubernetes120204](https://github.com/kubernetes/kubernetes/pull/120204), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Retry NodeStageVolume calls if CSI node driver is not running ([kubernetes/kubernetes120330](https://github.com/kubernetes/kubernetes/pull/120330), [rohitssingh](https://github.com/rohitssingh)) [SIG Apps, Storage and Testing]
- The kube-scheduler `selectorSpread` plugin has been removed, please use the `podTopologySpread` plugin instead. ([kubernetes/kubernetes117720](https://github.com/kubernetes/kubernetes/pull/117720), [kerthcet](https://github.com/kerthcet)) [SIG Scheduling]

API Change
- Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. ([kubernetes/kubernetes119807](https://github.com/kubernetes/kubernetes/pull/119807), [jpbetz](https://github.com/jpbetz)) [SIG API Machinery, Auth and Cloud Provider]
- Mark Job onPodConditions as optional in pod failure policy ([kubernetes/kubernetes120208](https://github.com/kubernetes/kubernetes/pull/120208), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]

API Change
- A CDIDevice field is included in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([kubernetes/kubernetes118254](https://github.com/kubernetes/kubernetes/pull/118254), [elezar](https://github.com/elezar)) [SIG Node and Testing]
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([kubernetes/kubernetes118420](https://github.com/kubernetes/kubernetes/pull/118420), [alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Added `ServedVersions` field to `StorageVersion` API. ([kubernetes/kubernetes118386](https://github.com/kubernetes/kubernetes/pull/118386), [Richabanker](https://github.com/Richabanker))
- Added `IP mode` field to loadbalancer status ingress. ([kubernetes/kubernetes118895](https://github.com/kubernetes/kubernetes/pull/118895), [RyanAoh](https://github.com/RyanAoh))
- Added `podReplacementPolicy` and terminating field to job api. ([kubernetes/kubernetes119301](https://github.com/kubernetes/kubernetes/pull/119301), [kannon92](https://github.com/kannon92))
- Added a new `namespaceParamRef` field to `admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy`. ([kubernetes/kubernetes119215](https://github.com/kubernetes/kubernetes/pull/119215), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Added a warning that TLS 1.3 ciphers are not configurable. ([kubernetes/kubernetes115399](https://github.com/kubernetes/kubernetes/pull/115399), [3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a `localhostProfile`. ([kubernetes/kubernetes117020](https://github.com/kubernetes/kubernetes/pull/117020), [cji](https://github.com/cji))
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([kubernetes/kubernetes118041](https://github.com/kubernetes/kubernetes/pull/118041), [cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([kubernetes/kubernetes118267](https://github.com/kubernetes/kubernetes/pull/118267), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Added new `CRDValidationRatcheting` alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([kubernetes/kubernetes118990](https://github.com/kubernetes/kubernetes/pull/118990), [alexzielenski](https://github.com/alexzielenski))
- Added new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([kubernetes/kubernetes118137](https://github.com/kubernetes/kubernetes/pull/118137), [helayoty](https://github.com/helayoty))
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([kubernetes/kubernetes115754](https://github.com/kubernetes/kubernetes/pull/115754), [linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Changed how KMS v2 encryption at rest can generate data encryption keys.
When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([kubernetes/kubernetes118828](https://github.com/kubernetes/kubernetes/pull/118828), [enj](https://github.com/enj))
- Exposed `rest.DefaultServerUrlFor` function. ([kubernetes/kubernetes118055](https://github.com/kubernetes/kubernetes/pull/118055), [timofurrer](https://github.com/timofurrer))
- Extended the Job API for alpha version of `BackoffLimitPerIndex`. ([kubernetes/kubernetes119294](https://github.com/kubernetes/kubernetes/pull/119294), [mimowo](https://github.com/mimowo))
- Graduated `AdmissionWebhookMatchCondition` feature to beta. ([kubernetes/kubernetes119380](https://github.com/kubernetes/kubernetes/pull/119380), [a-hilaly](https://github.com/a-hilaly))
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([kubernetes/kubernetes117793](https://github.com/kubernetes/kubernetes/pull/117793), [tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([kubernetes/kubernetes118782](https://github.com/kubernetes/kubernetes/pull/118782), [MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([kubernetes/kubernetes118883](https://github.com/kubernetes/kubernetes/pull/118883), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: added `--logging-format` flag to support structured logging. ([kubernetes/kubernetes117800](https://github.com/kubernetes/kubernetes/pull/117800), [cyclinder](https://github.com/cyclinder))
- NodeVolumeLimits implement the `PreFilter` extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([kubernetes/kubernetes115398](https://github.com/kubernetes/kubernetes/pull/115398), [tangwz](https://github.com/tangwz)) [SIG Scheduling]
- PersistentVolumes have a new `LastPhaseTransitionTime` field which holds a timestamp of when the volume last transitioned its phase. ([kubernetes/kubernetes116469](https://github.com/kubernetes/kubernetes/pull/116469), [RomanBednar](https://github.com/RomanBednar))
- Pods which set `hostNetwork: true` and declare ports, get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([kubernetes/kubernetes117696](https://github.com/kubernetes/kubernetes/pull/117696), [thockin](https://github.com/thockin)) [SIG Apps]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([kubernetes/kubernetes118644](https://github.com/kubernetes/kubernetes/pull/118644), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta, and it is turned off by default. ([kubernetes/kubernetes119409](https://github.com/kubernetes/kubernetes/pull/119409), [alexzielenski](https://github.com/alexzielenski))
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([kubernetes/kubernetes119264](https://github.com/kubernetes/kubernetes/pull/119264), [logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus`. ([kubernetes/kubernetes116335](https://github.com/kubernetes/kubernetes/pull/116335), [gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Removed `WindowsHostProcessContainers` feature-gate. ([kubernetes/kubernetes117570](https://github.com/kubernetes/kubernetes/pull/117570), [marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta. ([kubernetes/kubernetes117802](https://github.com/kubernetes/kubernetes/pull/117802), [kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([kubernetes/kubernetes119232](https://github.com/kubernetes/kubernetes/pull/119232), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([kubernetes/kubernetes117740](https://github.com/kubernetes/kubernetes/pull/117740), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- Supported `BackoffLimitPerIndex` in Jobs. ([kubernetes/kubernetes118009](https://github.com/kubernetes/kubernetes/pull/118009), [mimowo](https://github.com/mimowo))
- The `IPTablesOwnershipCleanup` feature (KEP-3178) is now GA; kubelet no longer
creates the `KUBE-MARK-DROP` chain (which has been unused for several releases)
or the `KUBE-MARK-MASQ` chain (which is now only created by kube-proxy). ([kubernetes/kubernetes119374](https://github.com/kubernetes/kubernetes/pull/119374), [danwinship](https://github.com/danwinship))
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([kubernetes/kubernetes117713](https://github.com/kubernetes/kubernetes/pull/117713), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([kubernetes/kubernetes117351](https://github.com/kubernetes/kubernetes/pull/117351), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([kubernetes/kubernetes116429](https://github.com/kubernetes/kubernetes/pull/116429), [gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Updated the comment about the feature-gate level for `PodFailurePolicy` from alpha to beta ([kubernetes/kubernetes118278](https://github.com/kubernetes/kubernetes/pull/118278), [mimowo](https://github.com/mimowo))
- `client-go`: Improved memory use of reflector caches when watching large numbers
of objects which do not change frequently. ([kubernetes/kubernetes113362](https://github.com/kubernetes/kubernetes/pull/113362), [sxllwx](https://github.com/sxllwx))
- `component-base/logs` is now stricter about not applying configurations multiple
times and will return an error when that is attempted. Can be overridden by binaries
which need to do that. ([kubernetes/kubernetes117108](https://github.com/kubernetes/kubernetes/pull/117108), [pohly](https://github.com/pohly))
- `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate
is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner`
controller loop removes service account token secrets that have not been used
in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting
to one year), **and are** referenced from the `.secrets` list of a ServiceAccount
object, **and are not** referenced from pods. ([kubernetes/kubernetes115554](https://github.com/kubernetes/kubernetes/pull/115554), [yt2985](https://github.com/yt2985))
- `kube-scheduler` component config (KubeSchedulerConfiguration) `kubescheduler.config.k8s.io/v1beta2`
is removed in `v1.28`. Migrate `kube-scheduler` configuration files to `kubescheduler.config.k8s.io/v1`. ([kubernetes/kubernetes117649](https://github.com/kubernetes/kubernetes/pull/117649), [SataQiu](https://github.com/SataQiu))
- Aggregated discovery now returns `responseKind: {}` for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. ([kubernetes/kubernetes119835](https://github.com/kubernetes/kubernetes/pull/119835), [liggitt](https://github.com/liggitt)) [SIG API Machinery and Testing]
- Fix CustomResourceDefinition status.storedVersions validation error messages. ([kubernetes/kubernetes119653](https://github.com/kubernetes/kubernetes/pull/119653), [sttts](https://github.com/sttts)) [SIG API Machinery]
- Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the `-v` command line flag when combined with `--config`. ([kubernetes/kubernetes119867](https://github.com/kubernetes/kubernetes/pull/119867), [pohly](https://github.com/pohly)) [SIG Network]
- PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. ([kubernetes/kubernetes116469](https://github.com/kubernetes/kubernetes/pull/116469), [RomanBednar](https://github.com/RomanBednar)) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing]
- Promoted API groups `ValidatingAdmissionPolicy` and `ValidatingAdmissionPolicyBinding` to `v1beta1`. ([kubernetes/kubernetes118644](https://github.com/kubernetes/kubernetes/pull/118644), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps and Testing]
- Promoted the feature gate `ValidtaingAdmissionPolicy` to beta and it is turned off by default. ([kubernetes/kubernetes119409](https://github.com/kubernetes/kubernetes/pull/119409), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing]
- Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the `KMSv2KDF` feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. ([kubernetes/kubernetes118828](https://github.com/kubernetes/kubernetes/pull/118828), [enj](https://github.com/enj)) [SIG API Machinery, Auth and Testing]
- A CDIDevice field is includes in the Device Plugin's `ContainerAllocateResponse`. This field maps to the CDIDevice field in the CRI protocol. ([kubernetes/kubernetes118254](https://github.com/kubernetes/kubernetes/pull/118254), [elezar](https://github.com/elezar)) [SIG Node and Testing]
- Add new annotation `batch.kubernetes.io/cronjob-scheduled-timestamp` to Job objects scheduled from CronJobs. ([kubernetes/kubernetes118137](https://github.com/kubernetes/kubernetes/pull/118137), [helayoty](https://github.com/helayoty)) [SIG Apps]
- Add podReplacementPolicy and terminating field to job api ([kubernetes/kubernetes119301](https://github.com/kubernetes/kubernetes/pull/119301), [kannon92](https://github.com/kannon92)) [SIG API Machinery and Apps]
- Added fields `reason` and `fieldPath` into CRD validation rules to allow users to specify reason and field path when validation failed. ([kubernetes/kubernetes118041](https://github.com/kubernetes/kubernetes/pull/118041), [cici37](https://github.com/cici37)) [SIG API Machinery]
- Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a `namespaceObject`
variable with expressions. ([kubernetes/kubernetes118267](https://github.com/kubernetes/kubernetes/pull/118267), [cici37](https://github.com/cici37)) [SIG API Machinery and Testing]
- Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. ([kubernetes/kubernetes118990](https://github.com/kubernetes/kubernetes/pull/118990), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy ([kubernetes/kubernetes119215](https://github.com/kubernetes/kubernetes/pull/119215), [alexzielenski](https://github.com/alexzielenski)) [SIG API Machinery and Testing]
- Extend the Job API for alpha version of BackoffLimitPerIndex ([kubernetes/kubernetes119294](https://github.com/kubernetes/kubernetes/pull/119294), [mimowo](https://github.com/mimowo)) [SIG API Machinery and Apps]
- Graduate `AdmissionWebhookMatchCondition` feature to beta ([kubernetes/kubernetes119380](https://github.com/kubernetes/kubernetes/pull/119380), [a-hilaly](https://github.com/a-hilaly)) [SIG API Machinery]
- In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . ([kubernetes/kubernetes118782](https://github.com/kubernetes/kubernetes/pull/118782), [MikeSpreitzer](https://github.com/MikeSpreitzer)) [SIG API Machinery]
- Indexed Job pods now have the pod completion index set as a pod label. ([kubernetes/kubernetes118883](https://github.com/kubernetes/kubernetes/pull/118883), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Kube-proxy: add '--logging-format' flag to support structured logging ([kubernetes/kubernetes117800](https://github.com/kubernetes/kubernetes/pull/117800), [cyclinder](https://github.com/cyclinder)) [SIG API Machinery, Architecture, Instrumentation and Network]
- Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to `BETA` stability. ([kubernetes/kubernetes119264](https://github.com/kubernetes/kubernetes/pull/119264), [logicalhan](https://github.com/logicalhan)) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
- Removed `resizeStatus` enum from `pvc.Status` and replaced with `AllocatedResourceStatus` ([kubernetes/kubernetes116335](https://github.com/kubernetes/kubernetes/pull/116335), [gnufied](https://github.com/gnufied)) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- StatefulSet pods now have the pod index set as a pod label `statefulset.kubernetes.io/pod-index`. ([kubernetes/kubernetes119232](https://github.com/kubernetes/kubernetes/pull/119232), [danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps]
- Support BackoffLimitPerIndex in Jobs ([kubernetes/kubernetes118009](https://github.com/kubernetes/kubernetes/pull/118009), [mimowo](https://github.com/mimowo)) [SIG API Machinery, Apps and Testing]
- Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver ([kubernetes/kubernetes117740](https://github.com/kubernetes/kubernetes/pull/117740), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
- The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
creates the KUBE-MARK-DROP chain (which has been unused for several releases)
or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). ([kubernetes/kubernetes119374](https://github.com/kubernetes/kubernetes/pull/119374), [danwinship](https://github.com/danwinship)) [SIG API Machinery, Network and Node]
- The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still `<pod>-<claim name>`, but a random suffix will avoid name collisions. ([kubernetes/kubernetes117351](https://github.com/kubernetes/kubernetes/pull/117351), [pohly](https://github.com/pohly)) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
- The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. ([kubernetes/kubernetes116429](https://github.com/kubernetes/kubernetes/pull/116429), [gjkim42](https://github.com/gjkim42)) [SIG API Machinery, Apps, Node, Scheduling and Testing]
- Add ServedVersions field to StorageVersion API ([kubernetes/kubernetes118386](https://github.com/kubernetes/kubernetes/pull/118386), [Richabanker](https://github.com/Richabanker)) [SIG API Machinery and Testing]
- Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. ([kubernetes/kubernetes117108](https://github.com/kubernetes/kubernetes/pull/117108), [pohly](https://github.com/pohly)) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]
- ACTION_REQUIRED
When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. ([kubernetes/kubernetes118420](https://github.com/kubernetes/kubernetes/pull/118420), [alculquicondor](https://github.com/alculquicondor)) [SIG Apps]
- Expose rest.DefaultServerUrlFor function ([kubernetes/kubernetes118055](https://github.com/kubernetes/kubernetes/pull/118055), [timofurrer](https://github.com/timofurrer)) [SIG API Machinery]
- If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via `memory.oom.group` . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. ([kubernetes/kubernetes117793](https://github.com/kubernetes/kubernetes/pull/117793), [tzneal](https://github.com/tzneal)) [SIG Apps, Node and Testing]
- Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([kubernetes/kubernetes118278](https://github.com/kubernetes/kubernetes/pull/118278), [mimowo](https://github.com/mimowo)) [SIG Apps]
- Added a warning that TLS 1.3 ciphers are not configurable. ([kubernetes/kubernetes115399](https://github.com/kubernetes/kubernetes/pull/115399), [3u13r](https://github.com/3u13r)) [SIG API Machinery and Node]
- Added error handling for seccomp localhost configurations that do not properly set a localhostProfile ([kubernetes/kubernetes117020](https://github.com/kubernetes/kubernetes/pull/117020), [cji](https://github.com/cji)) [SIG API Machinery and Node]
- Added new config option `delayCacheUntilActive` to `KubeSchedulerConfiguration` that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in `kube-scheduler` ([kubernetes/kubernetes115754](https://github.com/kubernetes/kubernetes/pull/115754), [linxiulei](https://github.com/linxiulei)) [SIG API Machinery and Scheduling]
- Client-go: Improved memory use of reflector caches when watching large numbers of objects which do not change frequently ([kubernetes/kubernetes113362](https://github.com/kubernetes/kubernetes/pull/113362), [sxllwx](https://github.com/sxllwx)) [SIG API Machinery]
- Kube-controller-manager: The `LegacyServiceAccountTokenCleanUp` feature gate is now available as alpha (off by default). When enabled, the `legacy-service-account-token-cleaner` controller loop removes service account token secrets that have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and are** referenced from the `.secrets` list of a ServiceAccount object, **and are not** referenced from pods. ([kubernetes/kubernetes115554](https://github.com/kubernetes/kubernetes/pull/115554), [yt2985](https://github.com/yt2985)) [SIG API Machinery, Apps, Auth, Release and Testing]
- Kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2 is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. ([kubernetes/kubernetes117649](https://github.com/kubernetes/kubernetes/pull/117649), [SataQiu](https://github.com/SataQiu)) [SIG API Machinery, Scheduling and Testing]
- NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. ([kubernetes/kubernetes115398](https://github.com/kubernetes/kubernetes/pull/115398), [tangwz](https://github.com/tangwz)) [SIG Scheduling]
- Pods which set `hostNetwork: true` and declare ports get the `hostPort` field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now `hostPort` will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. ([kubernetes/kubernetes117696](https://github.com/kubernetes/kubernetes/pull/117696), [thockin](https://github.com/thockin)) [SIG Apps]
- Removing WindowsHostProcessContainers feature-gate ([kubernetes/kubernetes117570](https://github.com/kubernetes/kubernetes/pull/117570), [marosset](https://github.com/marosset)) [SIG API Machinery, Apps, Auth, Node and Windows]
- Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta ([kubernetes/kubernetes117802](https://github.com/kubernetes/kubernetes/pull/117802), [kerthcet](https://github.com/kerthcet)) [SIG API Machinery and Apps]
- The `SelfSubjectReview` API is promoted to `authentication.k8s.io/v1` and the `kubectl auth whoami` command is GA. ([kubernetes/kubernetes117713](https://github.com/kubernetes/kubernetes/pull/117713), [nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Architecture, Auth, CLI and Testing]