On September 2, 2026, Kubernetes announced that scaling workloads to zero replicas with the Horizontal Pod Autoscaler (HPA) had reached Beta status in Kubernetes v1.37 and was enabled by default. For teams running expensive, intermittently used workloads, that opens the door to lower idle compute costs without relying entirely on external scaling controllers.
But there is a catch. Kubernetes can start scaling a workload back up when demand returns. That does not mean the first request will arrive at a ready application in time to meet its latency target. The distinction is easy to miss, and it is where many scale-to-zero designs run into trouble.
The real engineering challenge is to remove idle application capacity without making customers pay for the time it takes to bring that capacity back.
Why Scaling to Zero Can Break Latency SLOs
Consider an API that receives a few requests every hour. Keeping several Pods running around the clock wastes resources during long idle periods. Scaling to zero seems like the obvious fix.
Now imagine a customer opens the application after 30 minutes of inactivity. The incoming request reaches the cluster, but the application has no running Pods. Kubernetes must restore capacity before the request can be processed.
The delay can include metric detection, autoscaler reconciliation, scheduling, image retrieval, container startup, application initialization and readiness checks. If the cluster also needs a new node, infrastructure provisioning adds another delay.
A request can therefore wait seconds before the application is ready, even when the application normally responds in milliseconds.
This is why the Kubernetes v1.37 scale-to-zero announcement deserves attention alongside the latency problem it exposes. The feature enables scaling based on suitable object or external metrics, but the Kubernetes Service itself does not buffer HTTP requests while no backend Pods are ready.
What Happens During a Cold Start?
Cold-start latency is not a single operation. It is the accumulated delay between the arrival of demand and the application becoming capable of serving it.
For example, reducing the time required to detect demand will have limited value if a container takes several seconds to initialize. Likewise, a fast application startup cannot compensate for a long delay caused by unavailable cluster capacity.
I would investigate these stages separately rather than immediately changing autoscaler settings:
- Detection: How quickly does the metric or incoming request trigger activation?
- Scheduling: Is there a suitable node with enough CPU, memory or accelerator capacity?
- Startup: How much time goes into image pulls, application initialization and dependency checks?
- Routing: How long after readiness does the request reach the new backend?
Stage-level timing turns a vague cold-start complaint into a problem that can be measured and fixed.
Choose the Right Scaling Mechanism
Native Kubernetes HPA is a reasonable starting point for workloads that can wait for new capacity, particularly queue consumers and batch processors. With the v1.37 feature, an HPA can scale to zero using an appropriate object or external metric. CPU or memory utilization alone cannot drive scaling from zero because there are no running application Pods from which to obtain those utilization metrics.
The HPA scale-from-zero design proposal explicitly excludes request buffering at the Service level. That limitation is significant for public HTTP services, where the first request may need to remain open while the application starts.
For event-driven scaling, KEDA connects Kubernetes workloads to external event sources and metrics. It is useful when queue depth, message lag or another demand signal should determine whether workers run. KEDA core, however, does not automatically turn every HTTP request into a safely buffered activation event.
For HTTP workloads, the KEDA HTTP Add-on cold-start mechanism can hold requests while a backend scales up. Knative Serving offers another approach: its Activator handles traffic during scale-from-zero operations, while the Knative Pod Autoscaler adjusts replicas based on traffic and concurrency.
These systems can protect the request path, but none guarantees that a specific latency SLO will be met. The result still depends on how quickly the application starts and how much traffic arrives during that interval.
Buffering Helps, but It Does Not Make Requests Faster
Suppose an API has a hypothetical p99 latency SLO of 300 milliseconds, but its cold start takes two seconds. A request interceptor can hold the triggering request instead of letting it fail immediately because no backend is ready.
That improves the chance of a successful response, but the request still takes longer than two seconds to complete. It has avoided one type of failure and incurred a latency violation.
The KEDA HTTP Add-on supports pending-request limits and explicit overflow behavior. These safeguards matter because every held request consumes a connection and buffer space. An uncontrolled burst against a sleeping application can exhaust the interceptor itself.
Set a maximum number of pending requests, choose whether excess requests should be rejected or receive a placeholder response, and align the proxy timeout with the application’s expected startup time. A fallback service may also make sense when the original backend cannot become ready quickly enough.
Keep Capacity Warm Where the SLO Requires It
Not every workload should spend every quiet period at zero replicas. If the application must return a response within a few hundred milliseconds, retaining one ready Pod may be the simplest and most reliable option.
Knative supports positive minimum scale values, scale-down delays and controls for how long the final Pod can be retained. These settings let operators trade a limited amount of idle compute for a lower probability of encountering a cold start.
The retention period should reflect real traffic patterns. If requests normally arrive in bursts separated by short gaps, keeping a Pod alive through those gaps may avoid repeated startup costs. If the application remains idle for hours, the same policy may waste resources.
Predictive scaling is another option. Historical traffic, scheduled jobs and known upstream events can provide an early signal to start the application before demand arrives.
Prediction is not automatically better than a fixed retention period. It must be accurate enough, and provide sufficient lead time, to cover the workload’s actual startup duration.
Measure Latency and Cost Together
A useful scale-to-zero benchmark should compare an always-on baseline against several scaling policies: native HPA on external metrics, KEDA, an HTTP-aware activation layer, and a configuration that uses predictive prewarming.
Test sustained traffic, sudden bursts, short idle gaps, long idle periods and cold infrastructure. Repeat the experiments with realistic request concurrency. A test in which one request wakes one Pod will not reveal what happens when hundreds of requests arrive before the first Pod becomes ready.
Track p95 and p99 latency, timeouts, HTTP errors, cold-start duration, pending-request depth and time to readiness. Also record compute-hours saved, node-provisioning delays and the cost of retaining warm capacity.
Pay particular attention to requests that trigger activation. Aggregate latency can obscure their impact, especially when cold starts are infrequent or the SLO uses a high percentile. Record an explicit cold-start indicator where possible; the KEDA HTTP Add-on, for example, can expose the X-KEDA-HTTP-Cold-Start response header.
Optimize the Entire Activation Path
Once measurement reveals the bottleneck, optimize that stage. Reduce unnecessary initialization work, cache container images where appropriate, check readiness-probe behavior and keep suitable nodes available when provisioning delays dominate.
Do not treat application Pod count and cluster node count as the same thing. A workload can reach zero Pods while its nodes remain ready to schedule new ones. Scaling the node pool down as well may save more money, but it introduces additional provisioning time on the way back up.
The objective is not to maximize the time a workload spends at zero. It is to minimize compute cost while keeping latency and error rates within agreed limits.
Balancing Scale-to-Zero Savings Against Latency SLOs
Scaling Kubernetes workloads to zero can reduce idle compute costs, particularly for infrequently used workers and resource-intensive applications. But zero replicas should be treated as a capacity decision with measurable consequences, not as a free optimization.
For queue-based workloads, durable queues can absorb delays while workers start. For synchronous APIs with tight latency targets, warm capacity or a sufficiently fast alternative serving path may be necessary. Request buffering can improve availability, but it cannot erase a slow startup.
The right design comes from measuring the activation path, testing realistic traffic and comparing savings against SLO compliance. If a cold start takes longer than the request’s latency budget, the solution is to change the capacity strategy or the request contract, not simply to hope the autoscaler reacts faster.
