Zero-Downtime Deployments Explained: The Architecture, Health Checks & Connection Draining
· Sourabh G Kulkarni
Understand zero-downtime deployments: health checks, readiness probes, SIGTERM graceful shutdown, TCP connection draining, and load balancer synchronization.
In the early days of web operations, scheduled maintenance windows were a normal part of life.
Every Tuesday at 2:00 AM, companies would display a friendly maintenance splash screen: "Our site is down for routine upgrades. We'll be back online in two hours."
In today's global, 24/7 digital economy, maintenance windows are unacceptable. When an e-commerce platform processes millions of dollars an hour or a financial API handles hundreds of payment webhooks every second, dropping even 0.1% of active TCP connections means lost revenue, broken user sessions, and angry customers.
Achieving true Zero-Downtime Deployment (ZDD) is not just about using Kubernetes or spinning up containers. It is an intricate architectural choreography between readiness probes, graceful shutdown signals (SIGTERM), load balancer connection draining, and database schema evolution.
Here is how zero-downtime deployments actually work behind the scenes, and why so many deployments still drop connections despite claims of being zero-downtime.
1. Why Naive Deployments Drop Requests (The 502 Bad Gateway Mystery)
Consider what happens during a standard rolling update in Kubernetes when a pod is terminated:
[ Ingress Controller / ALB ]
│
├───> [ Pod 1: Terminating... ] ──x (Immediate hard kill? Connection dropped -> 502)
└───> [ Pod 2: Running v1.0 ]
When you update a Deployment to Version 2.0, two asynchronous events happen in parallel:
- Endpoint Controller Updates: The Kubernetes API server notifies the Ingress Controller /
kube-proxyto remove the old pod's IP address from the routing table. - Kubelet Sends Termination Signal: The kubelet on the worker node sends a
SIGTERMsignal to the container's PID 1 process.
The Race Condition
Updating IP routing tables across a distributed cluster takes time (often 1 to 3 seconds for iptables/eBPF rules or cloud load balancer targets to synchronize).
If your application process receives SIGTERM and immediately shuts down in 100 milliseconds, the load balancer is still forwarding active user requests to that pod's IP address for the next two seconds. Those users immediately receive an HTTP 502 Bad Gateway or Connection Refused error.
2. The Graceful Shutdown Protocol: Mastering SIGTERM
To prevent dropped requests, your application must handle Unix process termination gracefully.
When Kubernetes decides to terminate a pod, it follows a strict lifecycle:
1. preStop Hook Executes (Optional sleep)
│
2. SIGTERM Signal Sent to Container PID 1
│
3. Application Enters "Draining" Mode (Stop accepting new requests; finish existing ones)
│
4. terminationGracePeriodSeconds Timer (Default: 30s)
│
5. SIGKILL Sent (Force kill if app hasn't exited within grace period)
Implementing Graceful Shutdown in Application Code (Node.js Example)
const express = require('express');
const app = express();
const server = app.listen(3000);
let isShuttingDown = false;
// Health check returns 503 as soon as shutdown begins
app.get('/healthz', (req, res) => {
if (isShuttingDown) return res.status(503).send('Shutting down');
res.status(200).send('OK');
});
// Intercept SIGTERM from Linux / Kubernetes
process.on('SIGTERM', () => {
console.log('SIGTERM received. Starting graceful connection draining...');
isShuttingDown = true;
// Stop accepting new incoming TCP connections
server.close(() => {
console.log('All active HTTP requests finished. Closing database pool...');
db.pool.end(() => {
console.log('Database connections closed. Exiting process cleanly.');
process.exit(0);
});
});
// Failsafe timeout: Force exit if requests hang longer than 25 seconds
setTimeout(() => {
console.error('Graceful shutdown timed out. Forcing process exit.');
process.exit(1);
}, 25000);
});
3. The preStop Sleep Trick: Solving the Router Race Condition
Even with application-level graceful shutdown, there is still the distributed synchronization delay: the load balancer might send a request just after server.close() was called.
The industry-standard solution in Kubernetes is the preStop hook with a short sleep:
spec:
containers:
- name: api-server
image: myorg/api:v2.0
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
What This Accomplishes:
When Kubernetes marks the pod for termination:
- The
preStophook executessleep 5. - During those 5 seconds, the container continues running normally, serving requests.
- Simultaneously, Kubernetes removes the pod from the Service endpoints list, giving
kube-proxy, Cilium, and the Ingress Controller 5 full seconds to update their routing tables. - After 5 seconds, zero new requests are being routed to this pod.
- The kubelet then issues
SIGTERM, and the container safely drains remaining active connections with zero dropped requests.
4. Liveness vs. Readiness Probes: Don't Confuse Them
Kubernetes uses two critical health checks to manage traffic:
readinessProbe:
httpGet:
path: /ready
port: 3000
initialDelaySeconds: 5
periodSeconds: 3
livenessProbe:
httpGet:
path: /healthz
port: 3000
initialDelaySeconds: 15
periodSeconds: 10
- Readiness Probe ("Can I send you traffic?"):
Checks if the pod is initialized, warm, and ready to serve user requests (e.g., caches loaded, database migrations verified). If it fails, Kubernetes stops routing traffic to this pod without killing it. - Liveness Probe ("Are you alive or deadlocked?"):
Checks if the application is in an unrecoverable state (e.g., permanent deadlock or fatal memory corruption). If it fails, Kubernetes restarts the container.
Crucial Zero-Downtime Rule: Never point your Readiness probe to an external dependency (like your primary database or third-party payment gateway). If your database experiences a 2-second blip, all your pods will fail their readiness checks simultaneously, taking down your entire frontend routing layer.
5. The Database Challenge: Expanding and Contracting Schemas
The hardest part of zero-downtime deployments is not the application code—it is the database schema.
If Version 2.0 changes a database column name from user_name to full_name, you cannot simply run a migration that renames the column. During a rolling update, Version 1.0 and Version 2.0 run side-by-side. If you rename the column, Version 1.0 pods immediately crash.
The Expand and Contract Pattern
Always execute database migrations in phased releases:
- Phase 1 (Expand): Add the new column
full_namealongsideuser_name. Deploy an app release that writes to both columns but reads fromuser_name. - Phase 2 (Backfill): Run a background script to copy existing data from
user_nametofull_name. - Phase 3 (Switch): Deploy Version 2.0 to read and write exclusively from
full_name. - Phase 4 (Contract): Once all pods are stably running Version 2.0, safely drop the old
user_namecolumn.
Summary Zero-Downtime Checklist
maxUnavailable: 0: Never allow your rolling update to drop capacity below 100%.- Robust Readiness Probes: Ensure new pods only receive traffic after they are completely warmed up.
preStopSleep Hook: Add a 5–10 second preStop hook to allow load balancer routing tables to flush.SIGTERMInterception: Handle process termination signals to cleanly finish in-flight HTTP requests.- Phased Database Migrations: Use the Expand/Contract pattern to ensure database schemas support dual-version concurrency.