Kubernetes 1.37 matters less as a feature drop than as a control-plane decision point for platform teams standardising AI infrastructure. The immediate question is not whether gang scheduling exists, but whether native scheduling is now sufficient for your operating model. If your clusters mainly suffer from partial placement and stranded GPU capacity, the new PodGroup-based approach may let you simplify. If you also depend on tenant fairness, queue hierarchies, quota borrowing, or richer admission controls, native gang scheduling is still a building block rather than a replacement for tools such as Volcano or Kueue.
That makes rollout design important. Enabling GenericWorkload and the scheduling.k8s.io/v1alpha3 API introduces a new scheduling contract that application teams, batch platforms, and autoscaling policies all need to understand. In practice, operators should validate how minCount interacts with cluster autoscalers, priority classes, and admission workflows; otherwise jobs may wait cleanly but still wait too long because capacity expansion, quota policy, or preemption rules are not aligned with the scheduler’s new behavior.
The cost story is similarly more nuanced than “scale to zero saves money.” For GPU inference, the savings are real only if external metrics are reliable enough to wake workloads before request backlogs become user-visible. Teams should treat metric freshness, adapter availability, model image pull time, and accelerator initialisation as part of the production readiness checklist, because any weakness there turns idle-cost optimisation into an availability problem.
Seen together, these changes push Kubernetes toward workload-aware operations. The payoff is better GPU utilisation, but only for organisations willing to tighten the coupling between scheduling, autoscaling, and hardware lifecycle management.
The Cloud Native Computing Foundation shipped Kubernetes v1.37 on August 26, delivering the platform’s deepest scheduling overhaul in years — and surfacing a detail operators need to know before they update: gang scheduling, the release’s most significant AI-infrastructure addition, does not activate on its own. The new release, codenamed Garhwal after the Himalayan region in Uttarakhand, India that is home to release lead Dipesh Rawat, packages 67 enhancements across scheduling, networking, and security. Sixteen features graduated to Stable, 23 to Beta, and 27 entered Alpha — but two changes in the Beta column carry outsized implications for teams running AI training workloads on Kubernetes.
The Deadlock That Has Haunted Distributed AI Training
Anyone who has submitted a distributed training job requiring 64 GPUs across eight nodes knows how the failure unfolds: the scheduler places half the pods, then stalls because other workloads have claimed the remaining nodes. The job cannot run with half its workers. Partially scheduled pods sit idle, consuming cluster resources while blocking nothing useful. Engineers intervene manually.
Gang scheduling, formalized as KEP-4671 gang scheduling, addresses this at the platform level by guaranteeing all-or-nothing pod placement for a named group. The feature graduated to Beta in v1.37 and is defined via PodGroup and Workload objects in the scheduling.k8s.io/v1alpha3 API group. Operators configure placement guarantees using a minCount field — the minimum number of pods that must be schedulable simultaneously before any are placed. If the cluster cannot seat at least minCount pods from the group, none are bound, and the job waits cleanly in the unschedulable queue rather than consuming partial resources.
The critical operating detail: gang scheduling is disabled by default in v1.37. To activate it, operators must enable the GenericWorkload feature gate and the scheduling.k8s.io/v1alpha3 API group in their cluster configuration. The feature does not turn on with a standard upgrade to 1.37.
That minCount field also makes the feature more flexible than a strict "all pods or none" rule might suggest. For elastic training jobs — where the job can make progress with a subset of workers — operators can set minCount lower than the total desired pod count. The scheduler commits to placing at least minCount pods simultaneously; if more capacity is available, additional pods join the group. The field is mutable in v1.37, meaning operators can adjust it without recreating the PodGroup for already-running jobs.
Before this feature, teams managing distributed training clusters typically relied on external schedulers — Volcano, Apache YuniKorn, or NVIDIA’s KAI Scheduler — to implement gang scheduling behavior. Those tools remain relevant: they add quota borrowing, fair-sharing across tenants, and multi-queue management that native Kubernetes gang scheduling does not provide. For teams already running Volcano or Kueue, the v1.37 gang scheduling primitive is an alternative for the specific deadlock-prevention use case, not a wholesale replacement for the broader scheduling stack.
A related feature, workload-aware preemption (KEP-5710 preemption feature), also graduated to Beta. When a higher-priority training job needs to preempt lower-priority work, the scheduler now reasons about entire PodGroups rather than individual pods. This prevents the costly outcome of evicting half a distributed training job while leaving the other half running — consuming resources without making progress.
HPA Scale-to-Zero: GPU Cost Savings That Turn On Automatically
The economics argument for the 1.37 release is clearest in HPA scale-to-zero, which graduated to Beta with a meaningful distinction from gang scheduling: it is enabled by default for workloads that set spec.minReplicas: 0. The official Kubernetes blog documents the full feature specification.
First introduced as Alpha in Kubernetes v1.16, the feature allows a HorizontalPodAutoscaler to drive replica counts down to zero when demand disappears — and restore them when demand returns. For GPU-backed inference endpoints and batch processing queues, the potential cost reduction is significant: an inference service that sits idle overnight can fully release its GPU capacity rather than holding expensive instances in a reserved state.
The feature works through a narrow but well-defined mechanism. It operates only with external or object metrics — queue depth, a Prometheus custom metric, or any other external signal — because CPU and memory metrics require live pods to measure, and there are no live pods once replicas hit zero. When the HPA holds a workload at zero replicas, it records a ScaledToZero=True condition in its status, distinguishing an intentionally scaled-down workload from one where someone manually set the replica count to zero. Once the external metric crosses the configured threshold, the HPA scales back up and clears the condition.
"If you are running some high processing workloads on expensive GPUs, it’s going to save you some costs," Rawat told Network World in discussing the release.
The practical implication: teams operating inference-serving endpoints with predictable traffic patterns — peak during business hours, near-zero overnight — can set spec.minReplicas: 0 and configure a queue-depth external metric. The endpoint disappears when idle and reappears when requests queue up. Cold-start latency from zero (which can be significant for large model pods) is the primary tradeoff to evaluate before enabling scale-to-zero for latency-sensitive workloads.
DRA Device Taints: Quarantine a Bad GPU Without Draining the Node
Dynamic Resource Allocation, Kubernetes’ framework for assigning GPUs and other hardware accelerators, received its own Stable graduation in v1.37: device taints and tolerations (KEP-5055). The Kubernetes v1.37 release blog covers the full feature set.
The feature mirrors the node taint model that Kubernetes operators already know. A degraded or misbehaving GPU can be marked with a device taint — flagging it as unavailable for new workloads without removing it from the cluster or draining its entire node. New ResourceClaim objects that want to use the device can declare tolerations, allowing specific workloads to override the taint when necessary, such as a diagnostic job specifically targeted at the degraded device.
"The feature lets degraded hardware be drained using the familiar node taint model," said Sascha Grunert, principal engineer on the OpenShift Node team at Red Hat, in comments to Network World.
Administrators can also apply taints based on which driver manages a device, enabling targeted exclusion during driver maintenance windows — a common operational need in large GPU clusters where driver updates happen on rolling schedules.
Also graduating to Stable in this release is DRA Resource Claim Status with standardized network interface data (KEP-4817). This gives DRA drivers a consistent, structured way to describe attached network interfaces — relevant for GPU workloads that rely on RDMA fabrics for high-bandwidth inter-node communication during training runs. The Kubernetes v1.37 release blog describes the full DRA changes.
What Does Gang Scheduling Actually Do Under the Hood?
Gang scheduling’s mechanism is worth understanding precisely, because the all-or-nothing framing is an oversimplification.
When the GangScheduling plugin is active, pods belonging to a named PodGroup enter a PreEnqueue holding state. They do not move to the active scheduling queue until two conditions are met: the PodGroup object exists, and the number of pods created for the group equals or exceeds the configured minCount. Once that quorum is satisfied, the scheduler attempts to find placements for all unscheduled pods in the group in a single atomic scheduling cycle, using the PlacementFeasible extension point to track how many pods in the group have been successfully placed. The gang scheduling documentation describes the full scheduling cycle.
If the scheduler finds placements for at least minCount pods, those pods are bound to nodes. If it cannot satisfy the minCount requirement, no pods are scheduled — they return to the unschedulable queue to wait for resources to free up. This prevents the partial-placement deadlock without permanently blocking the cluster.
The architectural path to Beta was not linear. The v1.37 release consolidated two previously separate feature gates — GenericWorkload and a related gate from KEP-5832 — into a single GenericWorkload gate, simplifying the activation process for operators. The KEP-4671 gang scheduling README documents the consolidation. An Alpha feature also shipped in v1.37: hierarchical gang scheduling via CompositePodGroups, which allows an operator to define quorum requirements across groups of PodGroup objects — useful for multi-component workloads where entire subsystems must start together, not just individual pods.
Why Is Kubernetes Doing This? The Shift to Workload Awareness
Rawat described the architectural shift underway across the 1.35-1.37 release arc: "Previously it was just a pod and the scheduler depends on the pod and just schedules it. Now Kubernetes as a whole is trying to move towards being workload aware." He told Network World this shift reflects a fundamental change in the platform’s design philosophy.
The CNCF’s own framing of this moment is direct. CNCF executive director Jonathan Bryce noted that 66% of organizations already use Kubernetes as the operating system for AI. That adoption scale means the Kubernetes scheduler is now making resource-allocation decisions for a substantial fraction of global AI training compute — and it was doing so without native tools for the most basic property of distributed training jobs: that they require all workers to start together.
Gang scheduling, workload-aware preemption, and the earlier DRA GA (Kubernetes 1.34, August 2025) together represent a multi-release effort to make the Kubernetes scheduler reason about what operators actually need: not just "can this pod fit on a node?" but "can this entire job run, with all its parts, in a configuration that will actually complete work?"
Networking: iptables Is On the Way Out — Plan Your Migration
Alongside the AI scheduling story, v1.37 continues Kubernetes’ multi-release migration away from iptables and IPVS toward nftables for cluster traffic handling, with a concrete deadline operators need on their roadmaps.
Clusters that do not explicitly configure a kube-proxy mode will receive deprecation warnings as of v1.37 (KEP-5343). The IPVS backend is formally deprecated (KEP-5495), with a scheduled plan to disable it in Kubernetes 1.40 and remove it entirely in Kubernetes 1.43. The Kubernetes v1.37 release blog details the deprecation roadmap. Operators who have not begun this migration should start now.
The performance case for nftables is grounded in how each system handles rule updates at scale. iptables rewrites the entire ruleset on every change — a growing bottleneck in large clusters with thousands of services, where each service update triggers a full ruleset rewrite. nftables supports incremental rule changes: adding a new service rule is an O(1) operation rather than O(n). In clusters running 30,000 services, the Kubernetes nftables kube-proxy team’s benchmarks showed that nftables 99th-percentile first-packet latency was lower than iptables 1st-percentile latency — meaning the worst-case nftables cluster outperformed the best-case iptables cluster at that scale.
"nftables offers better performance through incremental rule updates and aligns with the direction of the Linux kernel networking stack," Grunert said.
Security: Native Workload PKI Without External Tools
Two security features reaching Stable in v1.37 give Kubernetes its first complete native public key infrastructure story for pod workloads.
Pod Certificates (KEP-4317) allow pods to request short-lived X.509 certificates directly from the Kubernetes API. The Kubernetes v1.37 release blog covers both features. ClusterTrustBundles (KEP-3257) provide a cluster-scoped mechanism to distribute trust anchors — root certificates — to pods through projected volumes. Together, the features make mutual TLS between services possible without external tooling like cert-manager or SPIFFE/SPIRE for basic use cases.
For teams operating zero-trust service meshes, the native PKI simplifies the simplest case — pod-to-pod mTLS within a single cluster — while more complex multi-cluster or multi-tenant identity scenarios may still benefit from SPIFFE-compatible systems. The Stable graduation means these APIs are now stable and will not break across Kubernetes versions.
Alpha: Pod-Level Checkpoint and Restore
One Alpha feature worth tracking for AI workloads is pod-level checkpoint and restore (KEP-5823). The Kubernetes Checkpoint/Restore Working Group announcement describes the initiative’s scope. The feature extends the Container Runtime Interface with CheckpointPod and RestorePod RPCs, enabling the kubelet and compatible container runtimes to snapshot a running pod and restore it later — potentially on a different node.
For long-running training jobs, this capability could eventually enable fault-tolerant training at the infrastructure level: checkpoint before a maintenance window, restore afterward, continue from where training left off rather than restarting from scratch. The feature is Alpha and requires runtime support from containerd or CRI-O; it is not production-ready. But the direction is clear, and the Kubernetes Checkpoint/Restore Working Group launched in January 2026 is specifically chartered to mature this capability.
What to Do Now
For teams running AI and ML workloads on Kubernetes, the practical near-term actions are:
Test gang scheduling in non-production environments. The GenericWorkload feature gate and scheduling.k8s.io/v1alpha3 API group must both be explicitly enabled. The PodGroup API is at scheduling.k8s.io/v1alpha3. Test with the appropriate minCount for your workloads before enabling in production.
Audit kube-proxy mode configuration. If no mode is explicitly configured, deprecation warnings will appear in v1.37. The IPVS mode will be disabled in Kubernetes 1.40. Evaluate nftables migration before that release to avoid forced migration under pressure.
Review DRA device taint policies for clusters with heterogeneous or aging GPU hardware. The device taint mechanism is now Stable and suitable for production use.
Evaluate HPA scale-to-zero for inference serving or batch inference queues. The feature is enabled by default in v1.37. Setting spec.minReplicas: 0 activates scale-to-zero behavior. Ensure external or object metrics (not CPU or memory) are configured to drive scale-up.
Full release notes and the complete changelog are available at the Kubernetes v1.37 release blog.
Frequently Asked Questions
Does gang scheduling automatically activate when I upgrade to Kubernetes 1.37?
No. Gang scheduling is Beta in v1.37 but ships disabled by default. To use it, you must explicitly enable the GenericWorkload feature gate and the scheduling.k8s.io/v1alpha3 API group in your cluster configuration. The HorizontalPodAutoscaler scale-to-zero feature, by contrast, is enabled by default — workloads that set spec.minReplicas: 0 will automatically be eligible for scale-to-zero behavior once they are upgraded to clusters running v1.37.
What is the difference between Kubernetes native gang scheduling and external tools like Volcano or Kueue?
Kubernetes 1.37’s native gang scheduling (KEP-4671) specifically solves the partial-placement deadlock: it guarantees that a named PodGroup is only scheduled when at least minCount pods can be placed simultaneously, preventing half-started distributed training jobs from consuming cluster resources. External schedulers like Volcano and Kueue go further — they add multi-tenant quota management, fair-sharing across teams, queue borrowing, and workload prioritization across queues. Teams with complex multi-tenant scheduling requirements will still need those tools; native gang scheduling is an alternative for the specific deadlock-prevention use case, not a replacement for a full scheduling stack.
When does IPVS support end in Kubernetes?
IPVS mode for kube-proxy is deprecated in Kubernetes v1.37, with planned disablement in Kubernetes 1.40 and full removal in Kubernetes 1.43. Operators who have not configured an explicit kube-proxy mode will see deprecation warnings after upgrading to v1.37. The recommended migration path is to nftables mode, which offers significantly lower rule-update latency in large clusters — particularly those running thousands of services.
What external metrics can trigger HPA scale-to-zero in Kubernetes 1.37?
HPA scale-to-zero works with external and object metrics, not CPU or memory. Common patterns include queue depth (from a message queue like Kafka or RabbitMQ), a custom Prometheus metric exposed by an application, or any other external signal available through a Kubernetes metrics adapter. The HPA scales replicas from zero when the metric crosses the configured threshold, and back to zero when the metric signals no pending work. Cold-start time — how long it takes a pod to become ready after scaling from zero — is the primary latency tradeoff and should be measured before enabling scale-to-zero for latency-sensitive endpoints.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.