← All writing

Zero-Downtime Deployments Explained: The Architecture, Health Checks & Connection Draining

· Sourabh G Kulkarni

Understand zero-downtime deployments: health checks, readiness probes, SIGTERM graceful shutdown, TCP connection draining, and load balancer synchronization.

In the early days of web operations, scheduled maintenance windows were a normal part of life.

Every Tuesday at 2:00 AM, companies would display a friendly maintenance splash screen: "Our site is down for routine upgrades. We'll be back online in two hours."

In today's global, 24/7 digital economy, maintenance windows are unacceptable. When an e-commerce platform processes millions of dollars an hour or a financial API handles hundreds of payment webhooks every second, dropping even 0.1% of active TCP connections means lost revenue, broken user sessions, and angry customers.

Achieving true Zero-Downtime Deployment (ZDD) is not just about using Kubernetes or spinning up containers. It is an intricate architectural choreography between readiness probes, graceful shutdown signals (SIGTERM), load balancer connection draining, and database schema evolution.

Here is how zero-downtime deployments actually work behind the scenes, and why so many deployments still drop connections despite claims of being zero-downtime.


1. Why Naive Deployments Drop Requests (The 502 Bad Gateway Mystery)

Consider what happens during a standard rolling update in Kubernetes when a pod is terminated:

[ Ingress Controller / ALB ]
          │
          ├───> [ Pod 1: Terminating... ] ──x (Immediate hard kill? Connection dropped -> 502)
          └───> [ Pod 2: Running v1.0 ]

When you update a Deployment to Version 2.0, two asynchronous events happen in parallel:

  1. Endpoint Controller Updates: The Kubernetes API server notifies the Ingress Controller / kube-proxy to remove the old pod's IP address from the routing table.
  2. Kubelet Sends Termination Signal: The kubelet on the worker node sends a SIGTERM signal to the container's PID 1 process.

The Race Condition

Updating IP routing tables across a distributed cluster takes time (often 1 to 3 seconds for iptables/eBPF rules or cloud load balancer targets to synchronize).

If your application process receives SIGTERM and immediately shuts down in 100 milliseconds, the load balancer is still forwarding active user requests to that pod's IP address for the next two seconds. Those users immediately receive an HTTP 502 Bad Gateway or Connection Refused error.


2. The Graceful Shutdown Protocol: Mastering SIGTERM

To prevent dropped requests, your application must handle Unix process termination gracefully.

When Kubernetes decides to terminate a pod, it follows a strict lifecycle:

1. preStop Hook Executes (Optional sleep)
       │
2. SIGTERM Signal Sent to Container PID 1
       │
3. Application Enters "Draining" Mode (Stop accepting new requests; finish existing ones)
       │
4. terminationGracePeriodSeconds Timer (Default: 30s)
       │
5. SIGKILL Sent (Force kill if app hasn't exited within grace period)

Implementing Graceful Shutdown in Application Code (Node.js Example)

const express = require('express');
const app = express();
const server = app.listen(3000);

let isShuttingDown = false;

// Health check returns 503 as soon as shutdown begins
app.get('/healthz', (req, res) => {
  if (isShuttingDown) return res.status(503).send('Shutting down');
  res.status(200).send('OK');
});

// Intercept SIGTERM from Linux / Kubernetes
process.on('SIGTERM', () => {
  console.log('SIGTERM received. Starting graceful connection draining...');
  isShuttingDown = true;

  // Stop accepting new incoming TCP connections
  server.close(() => {
    console.log('All active HTTP requests finished. Closing database pool...');
    db.pool.end(() => {
      console.log('Database connections closed. Exiting process cleanly.');
      process.exit(0);
    });
  });

  // Failsafe timeout: Force exit if requests hang longer than 25 seconds
  setTimeout(() => {
    console.error('Graceful shutdown timed out. Forcing process exit.');
    process.exit(1);
  }, 25000);
});

3. The preStop Sleep Trick: Solving the Router Race Condition

Even with application-level graceful shutdown, there is still the distributed synchronization delay: the load balancer might send a request just after server.close() was called.

The industry-standard solution in Kubernetes is the preStop hook with a short sleep:

spec:
  containers:
    - name: api-server
      image: myorg/api:v2.0
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

What This Accomplishes:

When Kubernetes marks the pod for termination:

  1. The preStop hook executes sleep 5.
  2. During those 5 seconds, the container continues running normally, serving requests.
  3. Simultaneously, Kubernetes removes the pod from the Service endpoints list, giving kube-proxy, Cilium, and the Ingress Controller 5 full seconds to update their routing tables.
  4. After 5 seconds, zero new requests are being routed to this pod.
  5. The kubelet then issues SIGTERM, and the container safely drains remaining active connections with zero dropped requests.

4. Liveness vs. Readiness Probes: Don't Confuse Them

Kubernetes uses two critical health checks to manage traffic:

readinessProbe:
  httpGet:
    path: /ready
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 3

livenessProbe:
  httpGet:
    path: /healthz
    port: 3000
  initialDelaySeconds: 15
  periodSeconds: 10
  • Readiness Probe ("Can I send you traffic?"):
    Checks if the pod is initialized, warm, and ready to serve user requests (e.g., caches loaded, database migrations verified). If it fails, Kubernetes stops routing traffic to this pod without killing it.
  • Liveness Probe ("Are you alive or deadlocked?"):
    Checks if the application is in an unrecoverable state (e.g., permanent deadlock or fatal memory corruption). If it fails, Kubernetes restarts the container.

Crucial Zero-Downtime Rule: Never point your Readiness probe to an external dependency (like your primary database or third-party payment gateway). If your database experiences a 2-second blip, all your pods will fail their readiness checks simultaneously, taking down your entire frontend routing layer.


5. The Database Challenge: Expanding and Contracting Schemas

The hardest part of zero-downtime deployments is not the application code—it is the database schema.

If Version 2.0 changes a database column name from user_name to full_name, you cannot simply run a migration that renames the column. During a rolling update, Version 1.0 and Version 2.0 run side-by-side. If you rename the column, Version 1.0 pods immediately crash.

The Expand and Contract Pattern

Always execute database migrations in phased releases:

  1. Phase 1 (Expand): Add the new column full_name alongside user_name. Deploy an app release that writes to both columns but reads from user_name.
  2. Phase 2 (Backfill): Run a background script to copy existing data from user_name to full_name.
  3. Phase 3 (Switch): Deploy Version 2.0 to read and write exclusively from full_name.
  4. Phase 4 (Contract): Once all pods are stably running Version 2.0, safely drop the old user_name column.

Summary Zero-Downtime Checklist

  1. maxUnavailable: 0: Never allow your rolling update to drop capacity below 100%.
  2. Robust Readiness Probes: Ensure new pods only receive traffic after they are completely warmed up.
  3. preStop Sleep Hook: Add a 5–10 second preStop hook to allow load balancer routing tables to flush.
  4. SIGTERM Interception: Handle process termination signals to cleanly finish in-flight HTTP requests.
  5. Phased Database Migrations: Use the Expand/Contract pattern to ensure database schemas support dual-version concurrency.