⎈ The Lookup That Only Failed Inside the Pod: A Kubernetes DNS War Story

Troubleshooting Kubernetes · Part 3 — start from Part 1 or see the full series →

Bring coffee, I said at the end of the last one. Bring patience. Bring a copy of the ndots documentation you’ve never actually read. I meant it as a joke — the kind of tired joke you make to a room that’s been on-call too long. It was not a joke. I want to be honest with you about that up front, because the version of me who wrote that sentence had no idea he was about to spend a Thursday afternoon proving it.

Welcome back to Troubleshooting Kubernetes. Today: the single most disorienting genre of bug in this whole business — the one where the same lookup works from one place and fails from another, three feet away, on the same network, at the same instant. If the last two war stories were about a system doing exactly what it was told, this one is about a system doing exactly what it was told and having that be the entire problem. You’ll see what I mean.

The message that ruined a good afternoon

Not a page this time. A Slack message, which is arguably worse, because a page at least has the decency to be an emergency. This was a developer, calm as anything: “hey, checkout can’t reach the payments API sometimes? like maybe 1 in 20 requests just times out.”

Sometimes. One in twenty. That word does something to a senior engineer that I can only compare to hearing a noise in your car that stops the moment you pull into the mechanic’s. Intermittent failures are where confidence goes to die. A thing that’s always broken is a gift — you can grab it, hold it, dissect it. A thing that’s broken one time in twenty is a ghost, and I have spent twenty-five years learning that ghosts are always, always real. You just haven’t found the floorboard yet.

The application logs were unambiguous about the symptom, at least:

dial tcp: lookup api.example.com on 10.96.0.10:53: read udp ... i/o timeout

lookup api.example.com ... i/o timeout. So it’s not the payments API. It’s not TCP. It’s not TLS. It’s DNS — the checkout pods, one time in twenty, simply could not turn api.example.com into an IP address. And DNS, as the saying goes, is never the problem, right up until it is the only problem you have ever had.

The three-terminal test

Here is the move that makes this story worth telling, and it’s the cheapest, dumbest move in the book: ask the same question from three different chairs.

From my laptop:

dig +short api.example.com
# 203.0.113.42   — instant

SSH’d onto the node the pod was running on:

dig +short api.example.com
# 203.0.113.42   — instant

And then, from inside the pod itself:

kubectl exec -it checkout-7c9f8-abcde -- sh
/ # time getent hosts api.example.com
# ...sometimes instant.
# ...sometimes a flat, horrible 5-second pause, then an answer.
# ...sometimes nothing at all. Timeout.

Same name. Same afternoon. Same physical network. My laptop: fine. The node the pod lives on: fine. The pod: haunted. I sat there for a second in the particular disorientation of that, the one where your model of reality has a hole in it, and I’ll be honest — a small, disgraceful part of me was delighted. This is the good stuff. This is why I still do this. Something was true that couldn’t be true, which means I didn’t understand something yet, which means I was about to.

resolv.conf tells the truth if you ask it

DNS on Linux is configured by one small, honest file, and a pod’s copy of it is different from a normal machine’s in ways that matter enormously. So I asked:

/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

There it is. options ndots:5. The floorboard.

Let me explain what that one line does, because it is the entire mystery and almost nobody who runs Kubernetes has actually read it. ndots:5 tells the resolver: “if the name someone asks for has fewer than 5 dots in it, don’t trust it — assume it’s a short, relative name that belongs to one of my search domains, and try those first.”

api.example.com has two dots. Two is fewer than five. So the resolver, being a faithful and literal machine — you’re sensing a theme in this series — does not go ask for api.example.com. Not first. First it walks the search list and asks CoreDNS, in order:

api.example.com.default.svc.cluster.local   → NXDOMAIN
api.example.com.svc.cluster.local           → NXDOMAIN
api.example.com.cluster.local               → NXDOMAIN
api.example.com                             → 203.0.113.42   (finally)

Four DNS round-trips to answer a question that my laptop answered in one, because my laptop has no ndots:5 and no cluster search domains. It just asks for the name it was given. The pod, by design, asks for four wrong names before the right one.

Why “sometimes”

Four extra queries is wasteful, but wasteful isn’t a timeout. The sometimes comes from something nastier living underneath. Each of those lookups fires a parallel pair of UDP packets — an A record and an AAAA record — from the same socket, same source port, same 5-tuple. Under the hood, the node’s NAT layer (conntrack) has a well-documented race inserting two entries for that identical tuple at the same instant; occasionally one of the replies gets dropped on the floor. And glibc’s resolver, when a UDP reply goes missing, doesn’t retry cleverly. It waits. Five seconds. Then tries again.

So: multiply “four doomed queries per external lookup” by “a UDP race that drops a packet every so often,” and once in a while a single api.example.com lookup eats a full five-second stall — which sails right past the checkout service’s HTTP client timeout, which surfaces to a customer as a failed request, which surfaces to me on a Thursday as sometimes. The node and my laptop never hit it because they resolve the name in one clean query and never go near the cluster search domains at all.

To watch it happen instead of just believing me, you point tcpdump at port 53 from inside the pod’s world and see the NXDOMAIN cascade with your own eyes:

kubectl exec -it checkout-7c9f8-abcde -- sh -c \
  'nslookup -debug api.example.com 2>&1 | grep -E "Name|NXDOMAIN|timed out"'

There they were. Three NXDOMAINs and, every so often, a connection timed out; no servers could be reached right in the middle of the cascade. The ghost had a floorboard, and the floorboard had a name, and the name was ndots.

Nothing was broken. That was the problem.

Here’s the part I love, and it is exactly the same shape as the volume that wouldn’t let go and the pod that lied: nothing here was broken. CoreDNS was healthy. The network was fine. The resolver was flawlessly, obediently doing what ndots:5 told it to do. And ndots:5 isn’t a mistake either — it’s a deliberate, sensible default. It exists so that when your app asks for payments — just payments, one word — it resolves to payments.default.svc.cluster.local in a single hop, because in-cluster service discovery is the common case and Kubernetes optimizes hard for it.

The default is tuned for the 90% of lookups that stay inside the cluster. And in doing that, it quietly taxes the 10% that reach out to the world — makes them slower, and, when the conntrack race bites, occasionally makes them fail. It’s not a bug. It’s a trade-off someone made for a different workload than yours, encoded in one line of a file you never open, executing perfectly, forever, until it’s 1-in-20 and it’s your Thursday.

The fix (and the fix behind the fix)

Two layers, and you want both.

Stop the bleeding — tell the pod to trust the name. For workloads that mostly talk to the outside world, override ndots on the pod spec so a name with a dot in it is tried as-is first:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

Now api.example.com (two dots ≥ 1) resolves in one hop, no search-domain cascade. Deploy that to the checkout pods and the 5-second stalls vanish immediately. The one caveat worth stating out loud: this changes how single-label names resolve, so if checkout also talks to a bare payments, test that it still finds it before you celebrate. Don’t cargo-cult ndots:1 across every workload in the cluster because a blog post (this one included) told you to — understand which side of the 90/10 each app lives on.

Fix it for real — kill the UDP race. The deeper cause is that conntrack packet drop, and the systemic answer is NodeLocal DNSCache: a per-node DNS cache that answers most queries locally over TCP, sidesteps the parallel-UDP conntrack race entirely, and takes a mountain of load off CoreDNS while it’s at it. If you’re seeing mysterious 5-second DNS delays anywhere in a cluster — and if you have more than a handful of nodes, you are, whether you’ve noticed or not — that’s the fix. And if CoreDNS itself is throwing SERVFAIL under all this pressure, that’s a related but separate rabbit hole; I wrote the CoreDNS SERVFAIL guide for the version of me who’ll be back here in a year having forgotten all of this.

Why I still love this

I did eventually get the tidy win — ndots:1 on the checkout deployment, NodeLocal DNSCache rolled out behind it, the 1-in-20 quietly becoming 0-in-a-lot. But that’s not the part I drove home thinking about.

I drove home thinking about ndots:5, and how it had been sitting in every pod’s resolv.conf in that cluster since the day it was built, doing its job with total integrity, and how not one person in the org could have told you it was there or what it did — including, until about 2 p.m. that Thursday, me. It wasn’t hidden. It wasn’t even complicated. It was just unread — a correct, deliberate decision made by someone I’ll never meet, for reasons that were good, that became a ghost the moment the workload it was tuned for stopped being the only workload in the room.

That’s the whole job, really. Not the fixing — the fix is knowable, it’s four lines of YAML. The job is reading the floorboards, learning what the faithful, literal machine was actually told, and having the humility to assume it’s doing precisely that before you assume it’s broken. It almost never is. It’s just honest, and honesty at scale is a strange and beautiful thing to debug. Twenty-five years in, an intermittent DNS timeout can still hand me an afternoon of pure, absorbed, disgraceful joy. I get paid for this. I’d probably pay for it.

Next time in Troubleshooting Kubernetes: the certificate nobody owned, that expired at 3:47 on a Sunday morning, and took the API server down with it. Bring coffee. Bring the runbook. Bring, if you can find it, whoever set the renewal reminder.

A per-node DNS cache and an ndots override will save you more 2 a.m. hours than almost anything else in Kubernetes. The running set of Kubernetes runbooks and error guides I keep — so I’m not re-deriving resolv.conf under pressure — lives on the Kubernetes toolkit. Read the floorboards.