Kubernetes SIGTERM: The Rolling Update That Cut In-Flight Requests

The Deployment rolled. New pods became Ready. Old pods went Terminating. Grafana showed a spike of 502s that lasted the length of the rollout, then vanished. The process logged Application is shutting down and dropped the request that was still on the wire. A rolling update sends SIGTERM to a pod the Service may still be using. The default grace period is a timer, not a drain.

The liveness post is which probe kills you while you are running. This is the order Kubernetes uses when it stops you, and the three numbers that have to add up or SIGKILL finishes the request.

What the kubelet does, in order

You kubectl rollout a Deployment, a ReplicaSet scales in, or a node drains. The API server sets deletionTimestamp. From that instant the pod is Terminating. Two things then happen at the same time:

1. The EndpointSlice controller drops the pod from the Service. kube-proxy, the cloud load balancer, and every ingress data plane have to notice. That is not a barrier the kubelet waits on.
2. The kubelet starts the termination sequence on the node.

The sequence on the node is:

preStop hook (if you defined one)
  → SIGTERM to PID 1
  → wait until the process exits, or until the grace period runs out
  → SIGKILL the container's cgroup

The grace period is terminationGracePeriodSeconds on the Pod spec. It defaults to 30. The countdown starts when the kubelet begins termination, before preStop runs. preStop is not extra time. It is the first slice of the same 30 seconds.

# The sleep is not in addition to 30. It is 15 of the 30.
terminationGracePeriodSeconds: 30
lifecycle:
  preStop:
    exec:
      command: ["sleep", "15"]

sleep 30 with a 30-second grace period means: when sleep returns, the timer is already at zero and the next signal is SIGKILL. The process never sees SIGTERM.

If the image is distroless, sleep is not there. preStop fails immediately, SIGTERM follows, and you are in the original race with a failed hook in the event log. Distroless needs a binary you shipped, or an httpGet preStop against a path the process already serves.

The race the sleep is for

EndpointSlice removal is a controller reconcile. kube-proxy programs iptables or nftables after it watches that change. An ingress controller, a service mesh sidecar, and a cloud NLB each have their own cache. SIGTERM does not wait for any of them.

So this happens, in production, on a cluster that is otherwise healthy:

t=0     pod marked Terminating
t=0     kubelet starts preStop / SIGTERM
t=0..3  a client still has this pod in its endpoint set
        POST /orders is in flight, or a keep-alive connection
        sends the next GET on the same socket
t=0.2   process received SIGTERM, closed the listener
        the in-flight POST gets ECONNRESET

preStop: sleep 15 exists to hold the process up and still serving while those data planes drop the pod. It is a delay, not a health check. Fifteen seconds is a common first value because EndpointSlice plus kube-proxy plus one ingress usually converge inside it. It is not a promise. A slow ingress, or an HTTP/1.1 keep-alive from a client that never saw the slice update, still lands on a Terminating pod.

The sleep only helps if the process keeps answering HTTP until SIGTERM. Do not start draining in preStop and in the SIGTERM handler unless you know which one owns the drain. There is no Kubernetes API the kubelet will block on that means "every kube-proxy has programmed this deletion."

SIGTERM has to reach the process that is listening

PID 1 receives the signal. If PID 1 is a shell, the Node or .NET process is a child.

# Shell form: /bin/sh is PID 1. SIGTERM may never reach node.
CMD npm start

# Exec form: node is PID 1. SIGTERM lands on the process you wrote.
CMD ["node", "server.js"]

command: ["sh", "-c", "node server.js"] in the Pod spec is the same trap. exec node server.js inside that shell replaces the shell. Without exec, you are waiting for the grace period to expire and then SIGKILL'ing a Node that never ran its handler.

A process that does receive SIGTERM still has to drain. Node's default with no listener is to exit. An HTTP server does not install a drain for you:

const server = app.listen(8080);

function drain(signal) {
  server.close((err) => {
    process.exit(err ? 1 : 0);
  });
  if (typeof server.closeIdleConnections === "function") {
    server.closeIdleConnections();
  }
}

process.on("SIGTERM", () => drain("SIGTERM"));
process.on("SIGINT", () => drain("SIGINT"));

server.close() refuses new connections and waits for existing ones. Keep-alive sockets count as existing. Without closeIdleConnections() (Node 18.2+) a client that holds the socket open holds the process open until SIGKILL.

ASP.NET Core's generic host already listens for SIGTERM and calls StopAsync. The catch is HostOptions.ShutdownTimeout, which defaults to five seconds, independent of the Pod's 30. Kestrel aborts what is still running at five seconds and the process exits "cleanly" while Kubernetes still had 25 seconds on the clock.

builder.Services.Configure<HostOptions>(options =>
{
    options.ShutdownTimeout = TimeSpan.FromSeconds(20);
});

That 20 has to fit in terminationGracePeriodSeconds minus preStop. If preStop sleeps 15 and the grace period is 30, the process has about 15 seconds of SIGTERM. A 20-second ShutdownTimeout is cut by SIGKILL. Set the host timeout below the remaining grace, not equal to the Pod spec.

Liveness during drain is a SIGKILL

The kubelet does not stop liveness probes because the pod is Terminating. If your SIGTERM handler closes the HTTP server first, the next /live fails. Enough failures and the kubelet kills the container. You do not get the rest of the grace period. A "graceful" shutdown lasts two seconds.

Keep /live up until you are actually exiting. Fail /ready as soon as you do not want new traffic — Terminating already told the EndpointSlice controller to drop you. The handler that SELECT 1s the database must not be on the liveness path here either: a drain plus a slow query is a failed probe plus SIGKILL.

livenessProbe:
  httpGet:
    path: /live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 1
terminationGracePeriodSeconds: 45
lifecycle:
  preStop:
    exec:
      command: ["sleep", "15"]

/live stays 200 while Kestrel or server.close() finishes in-flight work. /ready can go 503 the moment SIGTERM arrives. Do not lower liveness to failureThreshold: 1 and periodSeconds: 2 on a service you also drain; that turns one slow close into an immediate kill.

The three numbers have to fit

Budget	What it pays for	Typical mistake
preStop duration	Dataplane: EndpointSlice, kube-proxy, ingress	sleep longer than the grace period
Process drain (ShutdownTimeout, server.close)	In-flight requests after SIGTERM	Default 5s in ASP.NET, or no handler in Node
terminationGracePeriodSeconds	Both of the above, in one countdown	Left at 30 while preStop is 30

terminationGracePeriodSeconds  ≥  preStop  +  drain  +  a few seconds of slack

A 15-second preStop and a 20-second drain need a grace period of about 45, not 30. The Deployment's progressDeadlineSeconds and minReadySeconds do not extend this. maxUnavailable: 0 only waits until the new pod is Ready before it starts this sequence on the old one. It does not make SIGTERM wait for kube-proxy. A hung preStop.httpGet spends the whole budget before SIGTERM; keep the hook short.

What this is not

A liveness-versus-readiness mixup on a steady pod. That is the probes post. Here the pod is already dying. The probe mistake that matters is failing liveness because you started to die.

A slot swap. Azure App Service slots move routing at the front door. Kubernetes deletes endpoints and hopes every data plane noticed.

A cancelled GitHub Actions job. A cancelled runner is also SIGTERM, and concurrency is how you stop murdering a deploy job. It does not configure the cluster.

If the rollout is green and the 502s track old replicas down to zero, look at PID 1, the grace-period arithmetic, and whether /live still answers after SIGTERM. The Deployment replaced the pods. It did not drain the sockets you still had open.