The Illustrated Guide to Kubernetes Architecture: Control Planes, Worker Nodes, and Reconciliation Loops

When engineers first encounter Kubernetes, they often imagine it as a giant, magical operating system that runs Docker containers across a fleet of servers.

That mental model is not just incomplete—it leads to profound confusion the moment a cluster misbehaves.

Kubernetes does not actually run containers directly. It does not monitor containers with cron jobs, and it does not maintain persistent direct communication lines between its components.

Fundamentally, Kubernetes is a distributed, declarative state-reconciliation engine. You submit a specification of what the world should look like, and a set of independent, loosely coupled control loops work asynchronously to make reality match your declaration.

To truly understand Kubernetes—whether you are debugging an incident, preparing for a platform architecture interview, or designing an internal developer platform—you must understand how its components actually talk, who holds authority, and what happens under the hood during the lifecycle of a deployment.

1. The Macro Architecture: Two Realms

A Kubernetes cluster is divided strictly into two functional planes:

1. The Control Plane (The Brain): Maintains the cluster's state, evaluates health, takes scheduling decisions, and responds to cluster events.
2. Worker Nodes (The Muscle): The compute machines (virtual machines or bare metal) where your actual application containers live and execute.

┌────────────────────────────────────────────────────────────────────────┐
│                        THE CONTROL PLANE (BRAIN)                       │
│                                                                        │
│   ┌───────────────┐        ┌────────────────┐        ┌─────────────┐   │
│   │ kube-scheduler│ ◄────► │ kube-apiserver │ ◄────► │  kube-      │   │
│   └───────────────┘        └───────┬────────┘        │  controller-│   │
│                                    │                 │  manager    │   │
│                                    ▼                 └─────────────┘   │
│                             ┌─────────────┐                            │
│                             │    etcd     │ (Distributed Key-Value DB) │
│                             └─────────────┘                            │
└────────────────────────────────────┬───────────────────────────────────┘
                                     │ HTTPS / TLS gRPC
               ┌─────────────────────┴─────────────────────┐
               ▼                                           ▼
┌──────────────────────────────┐            ┌──────────────────────────────┐
│     WORKER NODE 01 (MUSCLE)  │            │     WORKER NODE 02 (MUSCLE)  │
│                              │            │                              │
│   ┌──────────────────────┐   │            │   ┌──────────────────────┐   │
│   │       kubelet        │   │            │   │       kubelet        │   │
│   └──────────┬───────────┘   │            │   └──────────┬───────────┘   │
│              ▼               │            │              ▼               │
│   ┌──────────────────────┐   │            │   ┌──────────────────────┐   │
│   │ Container Runtime    │   │            │   │ Container Runtime    │   │
│   │ (containerd / CRI-O) │   │            │   │ (containerd / CRI-O) │   │
│   └──────────┬───────────┘   │            │   └──────────┬───────────┘   │
│              ▼               │            │              ▼               │
│   ┌──────────────────────┐   │            │   ┌──────────────────────┐   │
│   │  [Pod A]   [Pod B]   │   │            │   │  [Pod C]   [Pod D]   │   │
│   └──────────────────────┘   │            │   └──────────────────────┘   │
│                              │            │                              │
│   ┌──────────────────────┐   │            │   ┌──────────────────────┐   │
│   │      kube-proxy      │   │            │   │      kube-proxy      │   │
│   └──────────────────────┘   │            │   └──────────────────────┘   │
└──────────────────────────────┘            └──────────────────────────────┘

The most important architectural rule in this entire system is this:

The Invariant: No component ever talks directly to etcd except kube-apiserver. Furthermore, worker nodes never talk to each other to coordinate scheduling. All coordination happens strictly through state updates emitted to and watched from the API server.

Let us inspect each component in depth.

2. Inside the Control Plane: The Four Pillars

1. kube-apiserver (The Front Door & Validator)

The API server is the central nervous system of Kubernetes. It exposes a JSON/Protobuf over HTTPS REST API. Everything you do with kubectl, every action an automated CI/CD pipeline takes, and every internal controller check flows through this binary.

When a request arrives at kube-apiserver, it passes through a strict sequential pipeline:

1. Authentication: Determines who you are (mTLS certificates, OIDC bearer tokens, Webhook tokens).
2. Authorization: Determines if you are allowed to perform the requested verb on the resource (primarily RBAC: Role-Based Access Control).
3. Mutating Admission Webhooks: Can modify the object before storage (e.g., automatically injecting an Istio sidecar container or setting default resource requests).
4. Schema Validation: Verifies the YAML conforms to the strict API schema.
5. Validating Admission Webhooks: Can reject the object based on security policies (e.g., blocking containers running as root via Kyverno or OPA Gatekeeper).
6. Storage: Only if every check passes, the object is serialized and written to etcd.

[ Request ] ──► [ AuthN ] ──► [ AuthZ ] ──► [ Mutating ] ──► [ Validate ] ──► [ etcd Write ]

2. etcd (The Single Source of Truth)

etcd is a strongly consistent, distributed key-value store based on the Raft consensus algorithm. It stores the complete specification and status of every resource in the cluster.

Key architectural characteristics:

• Consistent & Partition Tolerant (CP): Under the CAP theorem, etcd prioritizes consistency over availability. If a network partition prevents a cluster from achieving quorum (majority: $\lfloor n/2 \rfloor + 1$ nodes), writes are rejected to prevent split-brain state corruption.
• MVCC (Multi-Version Concurrency Control): It does not overwrite keys in place. Every modification increments a global 64-bit revision counter. This allows controllers to ask: "Tell me everything that changed since revision 40,201."
• Watch API: Instead of controllers polling etcd thousands of times per second (which would destroy CPU and network capacity), etcd streams change events over HTTP/2 gRPC streams directly to the API server.

3. kube-scheduler (The Placement Matchmaker)

The scheduler has one job: look for newly created Pods that have no assigned node (spec.nodeName == "") and pick the best worker node for them.

It does not launch the pod. It merely updates the Pod's spec.nodeName field via the API server.

The scheduling decision runs in two distinct phases:

[ Unscheduled Pod ]
        │
        ▼
┌──────────────────┐
│ 1. Filtering     │ ──► Discards nodes that cannot fit the pod
│    (Predicates)  │     (Insufficient CPU/RAM, NodeSelector mismatch, Taints/Tolerations)
└───────┬──────────┘
        │
        ▼
┌──────────────────┐
│ 2. Scoring       │ ──► Ranks remaining candidates 0–100
│    (Priorities)  │     (ImageLocality, LeastRequestedPriority, PodTopologySpread)
└───────┬──────────┘
        │
        ▼
[ Selected Node Binding ] ──► Writes assigned nodeName to API Server

1. Filtering (Predicates): Filters out nodes that fundamentally cannot host the pod. Checks:
   • Does the node have enough allocatable CPU and Memory?
   • Does the node match the nodeSelector or nodeAffinity rules?
   • Does the pod have tolerations for the node's taints?
   • Are requested host ports already bound?
2. Scoring (Priorities): Ranks the surviving nodes on a scale of 0 to 100 based on weighted algorithms:
   • LeastAllocatedPriority: Spreads pods across nodes to balance resource consumption.
   • ImageLocalityPriority: Favors nodes that already cached the container image locally to eliminate download delay.
   • PodTopologySpread: Spreads replicas across failure domains (racks or availability zones) to survive hardware failures.

4. kube-controller-manager (The Infinite Reconciliation Loop)

If the API server is the brain and etcd is the memory, the controller manager is the beating heart.

It is a single binary containing dozens of independent, concurrent control loops. Every controller implements the exact same fundamental logic:

$$\text{Reconcile}() = f(\text{Desired State}, \text{Actual State}) \longrightarrow \text{Action}$$

┌────────────────────────────────────────────────────────┐
│               THE CONTROL LOOP CYCLE                   │
│                                                        │
│       ┌─────────────────┐                              │
│       │  Observe State  │ ◄── Watch API Server         │
│       └────────┬────────┘                              │
│                ▼                                       │
│       ┌─────────────────┐                              │
│       │ Analyze Diff    │ ── (Desired != Actual?)      │
│       └────────┬────────┘                              │
│                ▼                                       │
│       ┌─────────────────┐                              │
│       │  Take Action    │ ── Create/Delete/Patch       │
│       └─────────────────┘                              │
└────────────────────────────────────────────────────────┘

A few critical built-in controllers include:

• DeploymentController: Watches Deployments. When a deployment changes, it creates or updates a underlying ReplicaSet.
• ReplicaSetController: Ensures the exact number of pod replicas declared in spec.replicas exist. If you asked for 3 and there are 2, it generates a new Pod object.
• NodeController: Monitors worker node heartbeats. If a node stops responding for more than pod-eviction-timeout (default 5 minutes), it schedules the evacuation of its pods.
• EndpointSliceController: Keeps network endpoints up to date as pods spin up or terminate, feeding live IP lists to kube-proxy.

3. Inside the Worker Node: The Execution Muscle

1. kubelet (The On-Host Captain)

The kubelet is an agent running natively as a systemd service on every worker node. It does not run inside a container.

Its primary responsibility is to watch for Pods assigned to its specific node, instruct the container runtime to launch them, continuously check their health probes (liveness, readiness, startup), and report node status back to the control plane.

Crucially, the kubelet does not know how to run Docker or containerd directly. Instead, it speaks to the container engine over a standardized Unix domain socket using gRPC: the Container Runtime Interface (CRI).

2. The Container Runtime (containerd / CRI-O & runc)

Modern Kubernetes does not use the full Docker daemon. Instead, it speaks directly to lightweight, compliant runtimes like containerd or CRI-O.

When kubelet issues a CRI gRPC call like RunPodSandbox or CreateContainer:

1. containerd receives the gRPC instruction.
2. It calls a low-level OCI runtime, typically runc.
3. runc interacts directly with the Linux kernel:
   • Sets up Linux Namespaces for isolation (pid, net, mnt, ipc, uts, user).
   • Sets up cgroups (Control Groups) for resource constraints (cpu.max, memory.max).
   • Mounts the root filesystem using an overlay storage driver (overlay2).
4. Once the process is spawned, runc exits, leaving a tiny supervisor process called containerd-shim to monitor the container without keeping runc resident in memory.

kubelet ──(gRPC CRI)──► containerd ──► containerd-shim ──► runc ──► Linux Kernel (cgroups/namespaces)

3. kube-proxy (The Network Plumber)

Pods in Kubernetes are ephemeral—they are created, destroyed, and rescheduled with completely new IP addresses constantly. You cannot point clients directly to pod IPs.

Kubernetes solves this with Services (stable virtual IPs). But who routes traffic from a Service IP (10.96.0.100) to the actual healthy Pod IP (172.16.2.45)?

That is the responsibility of kube-proxy.

kube-proxy runs on every node (typically as a DaemonSet). It watches the API server for Service and EndpointSlice updates and translates them into local network routing rules on the host OS:

• iptables mode (default): Translates service IPs by generating PREROUTING and KUBE-SERVICES chains in the Linux kernel netfilter. Packet routing uses random probabilistic load balancing.
• IPVS mode: Uses Linux IP Virtual Server (hash-table based), scaling efficiently to tens of thousands of services without the linear rule-traversal latency of iptables.
• eBPF (Cilium / modern alternatives): Bypasses kube-proxy and iptables completely, attaching eBPF programs directly to socket layers for zero-overhead packet routing.

4. The 10-Step Journey of a Deployment

To see how all these pieces interlock in real life, let us trace exactly what happens when you type:

kubectl apply -f web-deployment.yaml

[1. kubectl apply] ──► [2. API Server: AuthN/AuthZ/Admit] ──► [3. etcd write]
                                ▲
                                │ Watch Stream
                                ▼
               [4. Deployment Controller creates ReplicaSet]
                                │
                                ▼
               [5. ReplicaSet Controller creates 3 Unbound Pods]
                                │
                                ▼
               [6. kube-scheduler filters & scores nodes]
                                │
                                ▼
               [7. Scheduler binds Pod to Node-01 in etcd]
                                │
                                ▼
               [8. Kubelet on Node-01 receives Pod update]
                                │
                                ▼
               [9. Kubelet calls CRI -> containerd -> runc]
                                │
                                ▼
               [10. Pod is running; Endpoints updated via kube-proxy]

1. Client Submission: kubectl converts the YAML to JSON and issues an HTTP POST /apis/apps/v1/namespaces/default/deployments to kube-apiserver.
2. Admission & Persistence: The API server authenticates the caller, validates RBAC permissions, runs admission webhooks, and writes the Deployment record to etcd.
3. Deployment Controller Reacts: The DeploymentController (inside kube-controller-manager), which maintains an open watch on deployments, sees the new record. It generates a child ReplicaSet object and sends it back to the API server, which writes it to etcd.
4. ReplicaSet Controller Reacts: The ReplicaSetController detects its new ReplicaSet with spec.replicas: 3. It notices that 0 pods currently exist. It creates 3 distinct Pod objects with no node assigned (nodeName: "") and posts them to the API server.
5. Scheduler Steps In: The kube-scheduler watches for unbound pods. It detects the 3 pods. For each pod, it runs its Filtering and Scoring algorithms across available worker nodes, selects node-01, and issues a Binding API call setting pod.spec.nodeName = "node-01".
6. Kubelet Notices Its Assignment: The kubelet daemon running on node-01 maintains a watch on the API server for pods assigned to its node name. It discovers the new pod assignment.
7. Sandbox & Storage Allocation: kubelet tells the local Container Runtime (e.g., containerd) to create a Pod Sandbox (the network namespace and infrastructure container). It calls the Container Network Interface (CNI) plugin to assign a cluster IP to the pod.
8. Image Pull & Container Spawn: kubelet instructs the runtime to pull the container image if not cached locally, then invokes runc to spawn the application containers with assigned cgroup memory and CPU limits.
9. Health Probes Pass: kubelet monitors the pod's configured startupProbe and readinessProbe. Once the application returns HTTP 200, kubelet updates the Pod status to Ready via the API server.
10. Traffic Routing Enabled: The EndpointSliceController sees the Pod is now Ready and appends its IP to the matching Service endpoint list. On every worker node, kube-proxy immediately updates its local routing tables (iptables/IPVS/eBPF) so incoming traffic flows seamlessly to the new container.

5. Architectural Summary Table

The Big Takeaway

Kubernetes is often criticized for being overly complex. But that complexity is the deliberate result of a powerful design choice: extreme decoupling.

No single component knows the entire story.

• The scheduler does not know how to start a container.
• The controller manager does not know which physical node exists.
• The kubelet does not know why a pod was created or what deployment owns it.
• And etcd does not know what any of the data means.

By breaking container orchestration into independent, single-responsibility components communicating through a versioned REST API, Kubernetes achieves unmatched resilience. If the scheduler crashes, running pods keep serving traffic. If a worker node dies, control plane controllers detect the absence and seamlessly rebuild workloads elsewhere.

Once you visualize this dance of watchers, API records, and reconciliation loops, the cluster stops feeling like magic—and starts feeling like one of the most elegant distributed systems ever built.