Key takeaway: Orchestrators give a running process a deliberate window to finish its current work before forcibly killing it, and a service that does not use that window loses in-flight requests on every single deploy, restart and scale-down event, not just occasionally.
The Signal Most Services Never Handle
When an orchestrator decides to stop a running instance — during a deploy, a scale-down, or a node being drained for maintenance — it first sends a termination signal, giving the process a defined grace period to finish what it is doing and shut down cleanly, before sending a forceful kill signal if it has not exited by the end of that window.
A service with no explicit handling for the termination signal simply gets killed immediately once the grace period elapses, having done nothing during that window to protect in-flight work — any request being processed at the exact moment the signal arrives is abruptly cut off, and the client on the other end receives either an abrupt connection reset or, if particularly unlucky timing, no response at all with no indication of what happened to their request.
Why This Is Worse Than It Sounds at Small Scale
At low deployment frequency and low traffic, this might drop a request occasionally and go unnoticed, and it is not actually rare or occasional in any meaningful sense — it happens on literally every single deployment, every scale-down event, every node drain, which for a service deploying multiple times a day accumulates into a steady background rate of dropped requests that is entirely avoidable and simply has not been visible enough for anyone to trace back to this specific and fixable cause.
| Behaviour on termination signal | Result for in-flight requests |
|---|---|
| No handling at all | Abruptly killed when the grace period ends |
| Stop accepting new connections, finish in-flight ones | Cleanly completed within the grace period |
| Immediately close all connections on signal receipt | Also drops in-flight requests, just earlier |
What Correct Handling Actually Does
On receiving the termination signal, a correctly behaving service should immediately stop accepting new incoming connections or requests — so the orchestrator’s traffic routing has time to redirect new traffic elsewhere — while continuing to allow currently in-flight requests to complete normally, up to some reasonable internal timeout, before then exiting cleanly once those requests have finished or that timeout has elapsed.
def handle_sigterm(signum, frame):
server.stop_accepting_new_connections()
server.wait_for_in_flight_requests(timeout=25)
server.shutdown()The specific internal timeout should be set comfortably shorter than the orchestrator’s own configured grace period, so the process has genuinely finished its own clean shutdown sequence and exited on its own before the orchestrator’s forceful kill signal would otherwise arrive — a service racing right up against the grace period boundary risks still being forcefully killed anyway if its own cleanup takes even slightly longer than expected under load.
The Coordination With Load Balancer Deregistration
There is often a brief window between an orchestrator marking an instance for removal and the load balancer or service mesh actually stopping new traffic from reaching it, because that propagation is not instantaneous across a distributed system — a service that stops accepting connections the literal instant it receives the signal, without any brief delay accounting for this propagation lag, can still reject a handful of requests that were already routed to it before the routing layer caught up with the instance’s departure.
A short, deliberate delay after receiving the signal — continuing to accept requests normally for a few seconds specifically to bridge this propagation gap, before then actually beginning the stop-accepting-new-connections phase — closes this specific and easily overlooked edge case that a naive implementation, even one that otherwise handles the signal correctly, would still miss.
Verifying This Actually Works
Testing graceful shutdown deliberately — sending real traffic to an instance while manually triggering its termination, and confirming that in-flight requests genuinely complete successfully rather than being dropped — is the only way to actually verify this behaviour works as intended, since it will not show up as a failure in any normal test suite that does not specifically simulate a deploy happening mid-request.
The Bottom Line
Handle the termination signal explicitly: stop accepting new connections, allow in-flight requests to complete within a timeout shorter than the orchestrator’s grace period, and account for the brief propagation delay before load balancers stop routing new traffic to the departing instance. Test this behaviour deliberately by triggering a shutdown during active traffic, since silent request drops on every deploy will not surface in an ordinary test suite.
