← All writing

How DNS Actually Works Under the Hood: From Browser Keystroke to Resolvers, Caching & Anycast

· Sourabh G Kulkarni

Trace a DNS query under the hood: recursive resolvers, root servers, TLDs, authoritative nameservers, Anycast routing, TTL caching, and CoreDNS internals.

There is a famous running joke across sysadmins and DevOps engineers whenever an outage hits production:

"It's not DNS.
There's no way it's DNS.
It was DNS."

The Domain Name System (DNS) is the foundational phonebook of the entire internet. Every web request, database connection pool, API call, and microservice RPC hop begins with a domain name resolution. Yet because it operates transparently in the background, most engineers treat it as a black box: you buy a domain on Cloudflare or Route53, point an A record to an IP address, and forget about it.

Until an internal CoreDNS pod in your Kubernetes cluster hits CPU throttling, and suddenly every microservice in your cluster starts throwing connection timeout errors.

Here is the exact step-by-step journey of what happens across the wire when a machine resolves a domain name into an IP address.


1. The Step-by-Step Anatomy of a DNS Query

When you type api.example.com into your browser or curl an endpoint from a production server, your operating system does not immediately reach out to the root servers of the internet. It goes through a strict hierarchical chain.

[ Browser / App ]
       │  (1. Check local browser cache & /etc/hosts)
       ▼
[ OS Stub Resolver ]
       │  (2. Check OS DNS cache e.g. systemd-resolved)
       ▼
[ Recursive Resolver ] (e.g. 1.1.1.1, 8.8.8.8, or ISP)
       │  (3. Cache miss? Query Root Nameserver)
       ▼
[ Root Nameservers ] (13 logical root server addresses: a.root-servers.net ...)
       │  (4. "I don't know api.example.com, but .com is at 192.5.6.30")
       ▼
[ TLD Nameservers ] (.com TLD servers managed by Verisign)
       │  (5. "I don't know api, but example.com is at ns1.cloudflare.com")
       ▼
[ Authoritative Nameserver ] (Cloudflare / Route53 hosting your zone)
       │  (6. "api.example.com has A Record: 104.21.48.91 with TTL 300s")
       ▼
[ Destination IP Received ] ───> TCP Handshake (SYN -> SYN-ACK -> ACK) Begins

Phase 1: Local Resolvers (/etc/hosts & OS Cache)

  1. Application Cache: The browser or runtime checks its internal DNS cache.
  2. Operating System Stub Resolver: If not cached, the kernel inspects /etc/hosts (or C:\Windows\System32\drivers\etc\hosts). If no static mapping exists, it forwards the query to the nameservers defined in /etc/resolv.conf.

Phase 2: The Recursive Resolver (The Detective)

Your machine sends an outbound UDP packet on port 53 to a Recursive Resolver (such as Cloudflare 1.1.1.1, Google 8.8.8.8, or an AWS VPC Route53 resolver 10.0.0.2).

The recursive resolver's job is to chase down the answer on your behalf through the global DNS hierarchy:

  1. Root Nameservers (.): There are 13 logical root server IPs (named a.root-servers.net through m.root-servers.net), operated by organizations like ICANN, NASA, and Verisign. The root server does not know where api.example.com is, but it knows which nameservers handle the .com Top-Level Domain (TLD).
  2. TLD Nameservers (.com): The recursive resolver queries the .com TLD servers. The TLD server directs the resolver to the Authoritative Nameservers delegated for example.com (e.g., ns1.cloudflare.com).
  3. Authoritative Nameservers: This is where your actual DNS records live. Cloudflare or AWS Route53 answers: api.example.com -> 104.21.48.91 (TTL: 300).
  4. Caching & Return: The recursive resolver caches that answer for 300 seconds and returns the IP address to your machine.

2. Anycast: Why the 13 Root Servers Never Go Down

If there are only 13 root server IP addresses in the entire world, how does the global internet survive without these servers being knocked offline by DDoS attacks or natural disasters?

The answer is Anycast BGP Routing.

In traditional Unicast routing, one IP address maps to one physical server in one data center. If that server is in Virginia, every packet sent to that IP must travel across physical fiber optic cables to Virginia.

In Anycast routing, the exact same IP address is advertised by hundreds of data centers simultaneously across the globe using Border Gateway Protocol (BGP):

[ User in Tokyo ]      ────> (Nearest BGP Hop) ────> [ Root Server Node (Tokyo) ]
[ User in Frankfurt ]  ────> (Nearest BGP Hop) ────> [ Root Server Node (Frankfurt) ]
[ User in Mumbai ]     ────> (Nearest BGP Hop) ────> [ Root Server Node (Mumbai) ]

When you query 198.41.0.4 (a.root-servers.net), your internet service provider routes your packet to the topologically closest physical server running that address. Today, those 13 logical root addresses are backed by over 1,700 physical server clusters distributed worldwide. If a data center in London goes offline, BGP automatically re-routes packets to Amsterdam in milliseconds.


3. TTL (Time to Live) and the Cache Invalidation Myth

Every DNS record includes a TTL (Time to Live) value in seconds:

api.example.com.   300   IN   A   104.21.48.91
  • A TTL of 300 tells intermediate resolvers: "You may cache this IP and answer queries locally for 300 seconds (5 minutes) without asking me again."

The DevOps Trap: Emergency IP Migration

If your primary data center catches fire and you change an A record in Route53, your change does not take effect globally immediately.

Every recursive resolver on earth that queried your domain in the last 4 minutes will continue routing customer traffic to the dead IP address until their cached TTL expires.

The Best Practice:

  • Normal Operations: Set TTL to 3600 (1 hour) or 86400 (1 day) to reduce DNS lookup latency and save recursive DNS query costs.
  • Before Scheduled Migrations: 48 hours prior to migrating a database or changing cluster IPs, lower the TTL to 60 seconds. Once the migration is complete and verified, raise it back to 3600.

4. Kubernetes Internal DNS: The CoreDNS Bottleneck

Inside a Kubernetes cluster, DNS is handled by CoreDNS.

Every time a pod accesses another service via http://payment-service.default.svc.cluster.local, it queries CoreDNS over UDP.

Why Kubernetes Clusters Suffer DNS Outages

  1. The ndots:5 Problem: By default, Linux resolves short domain names by appending search domains defined in /etc/resolv.conf:

    search default.svc.cluster.local svc.cluster.local cluster.local
    options ndots:5
    

    If your application tries to access an external API like api.stripe.com, because it contains fewer than 5 dots, the Linux resolver queries:

    1. api.stripe.com.default.svc.cluster.local (NXDOMAIN)
    2. api.stripe.com.svc.cluster.local (NXDOMAIN)
    3. api.stripe.com.cluster.local (NXDOMAIN)
    4. api.stripe.com. (SUCCESS)

    Every single outbound API call generates 4 separate DNS queries to CoreDNS! In high-throughput clusters, this saturates CoreDNS worker threads and exhausts Linux conntrack tables.

How to Fix It

  • NodeLocal DNSCache: Deploy a DaemonSet that runs an in-memory caching DNS server on every worker node, preventing UDP packet storms from flooding the centralized CoreDNS pods.
  • Append Trailing Dots: In your application code, write external URLs with a trailing dot (https://api.stripe.com./) to tell the resolver it is a Fully Qualified Domain Name (FQDN), bypassing local cluster search domains instantly.

Summary Key Takeaways

  1. Hierarchy: Resolution flows through Browser -> OS -> Recursive Resolver -> Root -> TLD -> Authoritative Server.
  2. Anycast: DNS handles billions of requests with low latency by broadcasting identical IPs from hundreds of global points of presence.
  3. TTL Strategy: Lower your TTL 48 hours before major infrastructure migrations to ensure rapid propagation.
  4. Kubernetes Optimization: Enable NodeLocal DNSCache to protect CoreDNS from ndots:5 query multiplication in production.