The Kubernetes pod termination lifecycle looks straightforward in the documentation. Delete a pod, Kubernetes sends SIGTERM, the application shuts down cleanly, and after terminationGracePeriodSeconds (default 30 seconds) Kubernetes sends SIGKILL if the process is still running. Neat, sequential, predictable.
Except that is not what actually happens. Cloudflare’s engineering team documented a fix that recovered 600 hours of engineering time per year. The fix is a single line of YAML. The reason it works requires understanding what Kubernetes is actually doing when a pod terminates, and why distributed state propagation turns a seemingly simple operation into a window of failure.
What Happens When a Pod Is Deleted
When Kubernetes receives a pod deletion request, several things happen concurrently:
- The pod’s status transitions to
Terminating - The pod is removed from the
Endpoints(orEndpointSlice) object for every Service it backs - If a
preStophook is configured, it executes; otherwise, SIGTERM goes directly to the container - The grace period countdown begins
The word to focus on is concurrently. Steps 2 and 3 are initiated by the same control plane event at roughly the same time, but they are handled by different components with different propagation paths.
Removing the pod from Endpoints is an API server write. But the component that translates that write into actual traffic routing is kube-proxy, running as a DaemonSet on every node. kube-proxy watches the API server for endpoint changes and then updates iptables (or IPVS) rules on its node accordingly. On a cluster under any real load, this watch-react-update cycle takes time. Depending on cluster size, API server load, and the kube-proxy --iptables-sync-period (historically defaulting to 30 seconds, though more recent versions handle this more responsively), propagation can take anywhere from a few hundred milliseconds to several seconds.
Meanwhile, the container receives SIGTERM. If the application is well-behaved and fast, it might exit in 100-500 milliseconds. If it is very fast, it exits before kube-proxy has finished updating iptables on the nodes that are still routing traffic to it.
The result: a window where the pod is gone but traffic is still arriving. Every request that lands during that window gets a connection refused or a TCP reset.
[Pod deletion requested]
|
+---> Endpoints updated in API server
| ---> kube-proxy watches, sees change (1-5s later)
| ---> iptables updated on nodes
| ---> traffic finally stops routing here
|
+---> SIGTERM sent to container (immediately)
---> App exits cleanly (~100-500ms)
---> [gap: traffic still arriving, process is gone]At small scale or low deployment frequency, this is a minor annoyance. At Cloudflare’s scale, deploying many services across large clusters multiple times per day, every rolling update produces this window on every pod rotation, every replica at a time.
The Fix
The standard mitigation is a preStop lifecycle hook with a sleep:
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]That is the one-liner. Before Kubernetes sends SIGTERM to the container, it executes the preStop hook and waits for it to complete. The 5-second sleep gives kube-proxy time to observe the endpoint change and update iptables on every relevant node before the application starts its shutdown sequence.
The sequence becomes:
[Pod deletion requested]
|
+---> Endpoints updated in API server
| ---> kube-proxy propagating...
|
+---> preStop hook runs: sleep 5s
---> [5 seconds pass]
---> kube-proxy has updated iptables on nodes
---> no new traffic arriving
preStop completes
---> SIGTERM sent to container
---> App drains in-flight requests and exits cleanlyBy the time SIGTERM arrives, the routing layer has already stopped directing traffic to this pod. The application can drain whatever requests were in-flight when the preStop began and exit without causing connection errors.
Tuning the Sleep Duration
The right sleep value depends on cluster topology and kube-proxy configuration. With iptables mode and default sync periods on older clusters, worst-case propagation could approach the full sync interval. In practice on modern clusters with responsive watch-based updates, 2-3 seconds is usually sufficient. Five seconds is a conservative default that covers most environments without meaningfully extending deployment time.
The preStop duration counts against terminationGracePeriodSeconds. If your preStop sleeps for 5 seconds and your application needs up to 25 seconds to drain in-flight requests, your grace period needs to be at least 30 seconds. The default is exactly 30, which works out. If you run long-lived request workloads (streaming, large uploads, slow database queries), you may need to increase terminationGracePeriodSeconds and adjust accordingly.
With IPVS mode instead of iptables, kube-proxy applies changes more efficiently, particularly on large clusters where iptables rule sets become expensive to rewrite. Propagation is faster, and a shorter preStop sleep may be sufficient. But “faster” in distributed systems still means nonzero, and the preStop pattern remains correct.
Application-Level Shutdown Still Matters
The preStop sleep is a compensating control for propagation lag. It is not a substitute for the application handling SIGTERM correctly. Once SIGTERM arrives, the process needs to stop accepting new connections, finish processing any in-flight requests, and exit cleanly.
In Go, this typically means using the http.Server.Shutdown method with a context:
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
<-sigCh
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := server.Shutdown(ctx); err != nil {
log.Printf("shutdown error: %v", err)
}In Node.js:
process.on('SIGTERM', () => {
server.close(() => {
process.exit(0);
});
});An application that calls os.Exit(0) immediately on SIGTERM will drop in-flight requests regardless of how long the preStop sleep ran. The sleep prevents new requests from arriving. The application is responsible for completing the ones already in progress.
The Scale Arithmetic
Six hundred hours per year becomes legible when you trace the failure mode. Without the preStop sleep, every rolling update produces 5XX errors during pod rotation. Those errors hit alerting thresholds. An engineer investigates, confirms it is a deployment-related spike, and either waits for it to resolve or initiates a rollback. The incident gets logged. A short postmortem happens. Someone decides it is acceptable noise and moves on.
Multiply that by deployment frequency. A team shipping multiple services daily, each with multiple replicas cycling through rolling updates, generates dozens of these events per week. Even if each event takes only 15-20 minutes of engineering attention, the annual total climbs quickly into the hundreds of hours. At Cloudflare’s deployment scale, 600 hours in a year is conservative arithmetic, not hyperbole.
The fix costs nothing at runtime. The sleep happens during pod teardown, which is already on the critical path of the rolling update. Five seconds per pod rotation is invisible against the total deployment duration. The ROI is extreme.
Related Patterns in the Same Family
The termination race is the most common instance of a broader class of Kubernetes timing problems. The pattern appears in several other places.
Rolling updates where new pods start receiving traffic before their application has fully initialized. The fix is a properly configured readinessProbe: a probe that only returns 200 after the server is ready to handle requests, not merely after the process has started.
ConfigMap or Secret updates that take time to propagate to mounted volumes inside running pods. Applications that read configuration at startup are unaffected, but applications that watch for live config updates need to account for propagation lag across replicas.
Horizontal pod autoscaler scale-down events, where pods being removed during traffic reduction face the same endpoint propagation race as pods being removed during rolling updates. The preStop hook applies here as well.
In each case, the fix acknowledges that the Kubernetes control plane is eventually consistent by design. State changes propagate through a chain of watch-and-react loops, and the timing of any given propagation is bounded but not instantaneous. Writing correct operational configuration means accounting for that reality.
Why This Gets Missed
The Kubernetes documentation covers preStop hooks and graceful termination, but the connection between endpoint propagation timing and the need for a sleep delay is not prominently explained. Most teams discover it the same way Cloudflare did: by correlating deployment events with 5XX spikes in production metrics and tracing the cause.
The Kubernetes documentation on pod lifecycle describes the termination process, and the graceful shutdown best practices recommend preStop hooks, but the specific reasoning around kube-proxy propagation delay requires reading between the lines or finding a blog post from someone who already hit it.
The deeper point is that distributed systems require explicit reasoning about timing at every boundary. Kubernetes abstracts away a lot of infrastructure complexity, but it does not abstract away the physics of eventual consistency. A pod deletion request is not an atomic operation. It is a distributed transaction that completes in parallel across multiple components, and the components do not finish at the same time. One sleep statement is enough to account for that, once you understand why it is needed.