ARP and Neighbor Discovery on Linux

You have an IP address, you have a route, and yet a connection still fails with “No route to host.” Before you go digging through routing tables and firewall rules, there is a lower layer you have to rule out first: the one that turns an IP address into a physical hardware address so that a frame can actually be placed on the wire. That job belongs to ARP — the Address Resolution Protocol — and to the Linux neighbor table that caches its results. This lesson teaches you to read that table with ip neigh, to understand what each entry state is telling you, and to recognize the failures that look like routing problems but are really Layer 2.

You do not need deep networking theory to follow along. We will define ARP, the neighbor cache, and each state as we go, and we will keep tying every command back to a concrete operational question: what does this output tell us, and what should we test next?

What You Will Learn

• What ARP is and why every packet on a local network needs it
• How to read the Linux neighbor (ARP) cache with ip neigh
• What the states REACHABLE, STALE, FAILED and INCOMPLETE mean diagnostically
• Why ARP is a Layer 2, same-subnet-only protocol — and why that matters
• How duplicate IPs, wrong VLANs, and dead gateways show up in the neighbor table
• How to safely watch ARP traffic with tcpdump in a lab you own

Why IP Addresses Are Not Enough

An IP address is a logical address. It tells the network which host you mean, but not how to physically deliver a frame to that host on the local wire. That final delivery uses a MAC address (Media Access Control address) — a 48-bit hardware identifier assigned to each network interface, written like 52:54:00:a1:b2:c3.

So on any local network the kernel constantly answers one question: “I know the IP I want — what MAC do I send the frame to?” Resolving an IP address to its MAC address is exactly what ARP does.

Here is the flow when host 192.168.1.20 wants to talk to its gateway at 192.168.1.1:

192.168.1.20 wants 192.168.1.1

  "Who has 192.168.1.1?"   -->  broadcast to
                                the whole subnet

  Gateway replies:         <--  "192.168.1.1 is at
                                52:54:00:aa:bb:cc"

  192.168.1.20 caches:
    192.168.1.1  ->  52:54:00:aa:bb:cc

The requester shouts a question to every host in the broadcast domain (“Who has 192.168.1.1? Tell 192.168.1.20”). The one host that owns that IP answers with its MAC address. The requester caches the answer so it does not have to ask again for every single packet.

That cache is the neighbor table (historically called the ARP cache). Linux keeps one per interface, and you read it with ip neigh.

Reading the Neighbor Table with ip neigh

ip neigh

ip neigh (short for ip neighbour) prints the kernel’s neighbor cache — every IP-to-MAC mapping it currently knows, along with the interface it learned it on and a state. Typical output looks like this:

192.168.1.1 dev eth0 lladdr 52:54:00:aa:bb:cc REACHABLE
192.168.1.50 dev eth0 lladdr 52:54:00:11:22:33 STALE
192.168.1.99 dev eth0 FAILED

Read each line as a sentence: “To reach this IP, on this device, use this link-layer address (lladdr = the MAC), and here is how confident I am that the mapping still works.” That last word — the state — is the diagnostic gold. Everything useful about ARP troubleshooting comes from knowing what the state means.

🔎 Troubleshooting Tip — Do not just glance at the MAC address. The state is what tells you whether Layer 2 is healthy. A missing MAC with a FAILED state next to your gateway is a completely different problem than a STALE entry that just needs a nudge.

The Neighbor States That Matter

State	What it means	What it tells you
REACHABLE	Confirmed recently; the neighbor answered.	Layer 2 to this host is healthy right now.
STALE	Mapping is known but unconfirmed lately.	Usually fine — the kernel will re-verify on next use.
DELAY / PROBE	Kernel is actively re-checking the mapping.	Transient; it is deciding between REACHABLE and FAILED.
INCOMPLETE	ARP request sent, no reply yet.	Nobody has answered — often nobody is there.
FAILED	ARP resolution gave up; no answer.	The IP did not respond at Layer 2. Real problem.

The two to recognize instantly are INCOMPLETE and FAILED. Both mean the same thing operationally: the kernel asked “who has this IP?” and got silence. INCOMPLETE is still waiting; FAILED has given up. When either appears next to your default gateway, you have found your outage — and it is not a routing problem.

STALE looks alarming but almost never is. It just means “I have not confirmed this mapping in a while.” The moment you send traffic, the kernel re-verifies and flips it to REACHABLE. A STALE gateway entry is not a fault.

🛠️ DevOps Perspective — An INCOMPLETE/FAILED neighbor entry for the default gateway is the hidden cause behind a whole class of “No route to host” incidents. The application, the DNS name, and even the routing table all look correct — default via 192.168.1.1 is right there — but the kernel cannot resolve 192.168.1.1 to a MAC, so no frame can leave the interface. It looks like Layer 3 routing; it is really Layer 2 delivery. Always check ip neigh for the gateway before you blame the route.

Why ARP Only Works Within the Subnet

Here is the most important insight in this lesson: ARP is a Layer 2 protocol, and it only works inside your own subnet — your own broadcast domain. The “Who has 192.168.1.1?” question is a broadcast, and broadcasts do not cross routers. A router’s whole job is to stop broadcasts from flooding between networks.

That means you never ARP for a host on the other side of a router. If 192.168.1.20 wants to reach 10.0.5.30 on a different subnet, it does not ARP for 10.0.5.30 — that host is not on the local wire and would never hear the broadcast. Instead, the routing table says “to reach anything outside my subnet, hand the frame to the gateway,” so the host ARPs for the gateway’s MAC and sends there. The gateway forwards it on.

Same subnet:   ARP directly for the host's MAC
Other subnet:  ARP for the GATEWAY's MAC,
               let the router forward it onward

This is why your gateway’s neighbor entry is so critical. Every packet destined off your subnet — the internet, another VLAN, a database elsewhere — depends on resolving the gateway’s MAC first. If that one ARP resolution fails, everything beyond the local subnet becomes unreachable at once, even though local hosts still work fine.

🏭 Why This Matters in Production — When a cloud instance or Kubernetes node “loses the internet” but can still ping neighbors on the same subnet, suspect gateway ARP resolution. Local ARP succeeds host-to-host, but the gateway entry is FAILED, so nothing routes out. That points you straight at Layer 2 or the gateway itself instead of a wild goose chase through routing and firewall config.

What ARP Problems Actually Look Like

Because ARP sits so low in the stack, its failures masquerade as other things. A few signatures worth memorizing:

• Gateway entry is FAILED or INCOMPLETE — you cannot get off the subnet. Symptoms look like “No route to host” or a total loss of external connectivity while local hosts still respond.
• Duplicate IP addresses — two hosts share the same IP, so both answer the ARP request. The neighbor entry’s MAC may flip between two values and connections behave erratically. A mapping that keeps changing MACs means a duplicate IP.
• Wrong VLAN or broadcast domain — a host tagged for the wrong VLAN sends its ARP broadcasts to the wrong set of machines. The intended neighbor never hears the request, so the entry sits INCOMPLETE even though the IP addressing looks correct on paper.
• Layer 2 failures — a bad cable, a down switch port, or a misconfigured bridge means the broadcast physically never arrives. Again: INCOMPLETE, then FAILED.

The through-line: most of these show up as an INCOMPLETE/FAILED entry or a MAC that will not hold still. The neighbor table is your window into all of them.

🔎 Troubleshooting Tip — If a neighbor entry rapidly alternates between two different lladdr values, you almost certainly have two devices claiming the same IP. That is a duplicate-IP conflict, and no amount of routing or firewall tweaking will fix it — you have to find and re-address one of the two hosts.

Watching ARP on the Wire with tcpdump

The neighbor table shows you the result of ARP. Sometimes you want to watch the conversation itself — the actual requests and replies — to see who is answering. In a lab you own, tcpdump filtered to ARP does exactly that.

tcpdump -i eth0 arp

tcpdump -i eth0 arp captures on interface eth0 and, thanks to the arp filter, shows only ARP frames — nothing else. A healthy request-and-reply pair looks like this:

ARP, Request who-has 192.168.1.1 tell 192.168.1.20
ARP, Reply 192.168.1.1 is-at 52:54:00:aa:bb:cc

The first line is the broadcast question; the second is the answer (“192.168.1.1 is-at …”). If the request goes out but no reply comes back, you have proven the neighbor is not answering — the exact cause behind an INCOMPLETE/FAILED entry. And two different replies for the same IP means you have caught a duplicate-IP conflict in the act.

🔐 Security Note — Only capture, scan, or inspect traffic on systems you own or have explicit permission to assess. Packet capture can expose other users’ traffic, so run these examples in a controlled lab or on reader-owned infrastructure. Frame this as authorized infrastructure validation and defensive troubleshooting — never as an attack technique. For a deeper, safe walkthrough of capture filters, see tcpdump packet analysis.

Try It Yourself

🧪 Try It — Build the muscle memory on a machine you control:

1. Run ip neigh and read each line as a sentence: IP, device, MAC, state. Identify the default gateway (cross-check ip route — the default via address).
2. ping -c1 <gateway-ip>, then ip neigh again. The gateway entry should now read REACHABLE.
3. Leave the machine idle a few minutes and watch entries drift to STALE. Send one packet and see them flip back to REACHABLE — proof STALE is normal, not a fault.
4. Run tcpdump -i eth0 arp in one terminal; in another, ping a fresh local IP. Watch the who-has / is-at pair appear as the kernel resolves it.

Common Problems

• “No route to host” but the route exists. Check the gateway in ip neigh. An INCOMPLETE/FAILED gateway entry means Layer 2 to the gateway is broken — the route is fine, the delivery is not.
• External traffic dead, local traffic fine. Classic gateway-ARP failure. Local hosts respond; the gateway does not, so nothing leaves the subnet.
• Connections erratic / MAC keeps changing. Suspect a duplicate IP. Capture with tcpdump ... arp to see the competing replies.
• Entry stuck INCOMPLETE. Nobody is answering — target down, wrong VLAN, or a Layer 2 fault (cable/switch/bridge). Not a routing fix.
• Treating STALE as a fault. Do not “fix” it — it re-verifies automatically on the next packet.

Troubleshooting Workflow

Work the layers bottom-up when a connection fails and you suspect the low levels:

Application -> TLS -> Port -> DNS -> Gateway -> Route -> Interface
                                        ^
                              ip neigh checks THIS:
                              can we resolve the
                              gateway's MAC at all?

1. Observe — the exact error (“No route to host”) and whether local hosts still respond.
2. Hypothesis — local works but external does not → suspect gateway ARP.
3. Test — ip neigh for the gateway entry; ip route confirms which IP is the gateway.
4. Evidence — gateway shows FAILED/INCOMPLETE; tcpdump -i eth0 arp shows requests with no reply.
5. Identify the layer — Layer 2 neighbor resolution, not Layer 3 routing.
6. Correct — verify the gateway is up, the host is on the right VLAN/subnet, and no duplicate IP exists.
7. Validate — ping the gateway, confirm REACHABLE, then confirm external connectivity is back.

For the routing side of this picture, review Linux routing; to turn these findings into a repeatable connectivity check, see testing connectivity.

What You Learned

• ARP resolves a logical IP address to a physical MAC address so a frame can be delivered on the local wire.
• The Linux neighbor table, read with ip neigh, caches those IP-to-MAC mappings per interface, each with a state.
• REACHABLE = confirmed healthy; STALE = known but unconfirmed (usually fine); INCOMPLETE = asked, still waiting; FAILED = asked, no answer (a real Layer 2 problem).
• ARP is Layer 2 and works only within your own subnet/broadcast domain — you never ARP across a router; you ARP for the gateway and let it forward.
• A FAILED/INCOMPLETE gateway entry means you cannot get off the subnet — the cause of many “No route to host” symptoms that only look like routing.
• Duplicate IPs, wrong VLANs, and Layer 2 faults all surface as bad neighbor entries or a flip-flopping MAC.
• tcpdump -i eth0 arp lets you watch who-has / is-at exchanges directly — but only on systems you own or are authorized to test.

Next up in Part 6, we move from delivering frames to the transport layer: TCP vs UDP — how connections are established, why one protocol guarantees delivery and the other does not, and what each choice means when you are troubleshooting real services.