🧭 Debugging DNS Resolution with systemd-resolved on Linux

The single most confusing change to land on Linux servers in the last decade is systemd-resolved. One day /etc/resolv.conf was a file you edited by hand; the next it was a symlink to something in /run, your edits vanished on reboot, and nslookup and the actual application disagreed about what an A record pointed to. I have lost more hours than I’d like to admit to this stub resolver, so here’s how it actually works and how to debug it.

What systemd-resolved actually does

systemd-resolved runs a local DNS stub listener, usually on 127.0.0.53:53. Applications that use glibc’s NSS go through it, and it forwards queries to your real upstream servers. It also caches responses, handles split-DNS per interface, and (optionally) does DNSSEC validation.

The key mental model: there are now two places DNS config lives. The classic /etc/resolv.conf is for legacy clients, and resolved’s own per-link configuration is the real source of truth. When those two disagree, you get the classic “works in dig, breaks in the app” bug.

Step one: figure out who is answering

Before anything else, confirm which resolver path you’re on.

# Is resolv.conf the stub symlink?
ls -l /etc/resolv.conf
# /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

cat /etc/resolv.conf
# nameserver 127.0.0.53   <- the stub

If nameserver 127.0.0.53 is there, every glibc lookup is going through resolved. Now ask resolved itself what it thinks:

resolvectl status

This is the command I run first, every time. It shows you, per interface, the current DNS servers, the search domains, whether DNSSEC is on, and which link is the default route for DNS. If your VPN interface has DNS servers and your primary doesn’t, that’s where split-DNS surprises come from.

Step two: query through the right tool

dig and nslookup talk to whatever is in resolv.conf (or a server you name explicitly). resolvectl query talks to resolved the same way your application does. Use the latter to reproduce real failures:

resolvectl query api.internal.example.com

The output tells you which protocol answered, whether it came from cache, and which link served it. Compare that to a direct query that bypasses the stub:

dig @10.0.0.2 api.internal.example.com

If dig @upstream works but resolvectl query fails, the problem is in resolved’s configuration or routing — not your upstream server. That single comparison eliminates half the possible causes immediately.

Step three: check the per-link config

Split-DNS lives in per-link settings. List them:

resolvectl dns      # DNS servers per link
resolvectl domain   # routing/search domains per link

A “routing domain” (shown with a ~ prefix, like ~corp.example.com) tells resolved “send queries for this domain to this link’s servers only.” If an internal zone isn’t resolving, it’s often because no link claims that routing domain, so the query goes to your public upstream which returns NXDOMAIN.

To fix it for an interface managed outside NetworkManager, you can set it directly:

sudo resolvectl dns eth1 10.0.0.2
sudo resolvectl domain eth1 '~corp.example.com'

For persistence, put this in the network management layer that owns the link — systemd-networkd .network files, NetworkManager connection profiles, or netplan — not in /etc/resolv.conf, which will get clobbered.

Step four: cache and DNSSEC gotchas

Two resolved behaviors bite people repeatedly.

Stale cache. After you change a record upstream, the local cache can serve the old answer until TTL. Flush it:

sudo resolvectl flush-caches
resolvectl statistics   # confirm cache size dropped

DNSSEC false failures. If an upstream or a captive portal mangles responses, strict DNSSEC validation turns a working name into SERVFAIL. Check whether validation is the cause:

resolvectl query --validate=no problem.example.com

If that succeeds and the normal query fails, DNSSEC is your culprit. The right fix is usually a better upstream, but DNSSEC=allow-downgrade in /etc/systemd/resolved.conf is the pragmatic middle ground for messy networks.

Step five: watch it live

When intermittent failures won’t reproduce on demand, turn up logging and watch:

sudo resolvectl log-level debug
journalctl -u systemd-resolved -f

You’ll see each transaction, which server it went to, and why it failed. Set the level back to info when you’re done so you don’t fill the journal.

A debugging checklist worth saving

When DNS “randomly” breaks on a resolved box, walk this in order:

1. ls -l /etc/resolv.conf — stub or static?
2. resolvectl status — right servers and domains per link?
3. resolvectl query X vs dig @upstream X — stub problem or upstream problem?
4. resolvectl domain — does any link route the failing zone?
5. resolvectl flush-caches — stale answer?
6. --validate=no — DNSSEC false negative?

Nine times out of ten you find it before step five.

Where AI helps

The slow part of resolved debugging is interpreting the wall of resolvectl status output across many interfaces and correlating it with journalctl traces. That’s a pattern-matching task AI is good at: paste the status output and the failing query, and ask which link should be answering the zone and isn’t. I keep a small set of Linux admin prompts for exactly this kind of “explain this config and tell me what’s misrouted” work.

The tooling shifted under us with resolved, but the discipline didn’t: figure out who answers, query through the same path the app uses, and compare against a direct upstream query. Do that and the stub resolver stops being a mystery.

Generated commands and configs are assistive, not authoritative. Always verify against your own systems before applying changes in production.