Traceroute, MTR, and Network Path Analysis

When a connection is slow or flaky and the two ends are on different networks, the interesting question is rarely “is the destination up?” — it’s “where along the way does the traffic start to struggle?” A packet from your host to a remote service crosses a series of routers, and any one of them, or the link between two of them, can be the problem. Path analysis turns “the internet is slow” into “loss starts at hop 6, which is our edge firewall.” This lesson teaches traceroute and mtr: what a hop is, how these tools reconstruct the path, how to read round-trip time and packet loss — and, just as important, how to not over-read them, because a raw traceroute is one of the easiest tools to misinterpret.

Everything here is diagnostic and read-only, but tracing still sends probe packets across every network between you and the target.

🔐 Security Note — Only scan, inspect, or test systems you own or have explicit permission to assess. Trace only to destinations you own or are authorized to test; a traceroute crosses every intermediate network, so keep targets to your own infrastructure and reader-owned labs. This is authorized infrastructure validation and defensive troubleshooting, not hacking.

What You Will Learn

• What a hop is and how traceroute uses incrementing TTL to discover each one
• How to read round-trip time (RTT) and packet loss per hop
• Why a * (asterisk) does not always mean a hop is failing
• Why the return path can differ from the forward path (asymmetric routing)
• Why latency at one hop that clears at the next isn’t that hop’s fault
• How mtr improves on a single traceroute for intermittent problems
• How to use path analysis to separate “my network” from “somewhere upstream”

Hops, Routers, and the Path

A hop is one router along the way to a destination. Your packet leaves your host, reaches your default gateway (the first hop), which forwards it to the next router, and so on until it arrives. Each forwarding device is a hop; the ordered list of hops is the path.

Conceptually, a request from a workstation to a public service looks like this:

Host
  │
  ▼
Gateway (your LAN router)
  │
  ▼
Core Router (your site / ISP edge)
  │
  ▼
Firewall
  │
  ▼
Internet (many carrier hops)
  │
  ▼
Destination

Path analysis walks this chain and measures each step. If everything is healthy up to the firewall and then loss appears on every hop beyond it, you’ve localized the change to that boundary — far more actionable than “packets are dropping somewhere.”

How Traceroute Actually Works (TTL)

Traceroute is a clever trick built on a field that every IP packet already carries: TTL (time to live). TTL is a counter that starts at some value and is decremented by one at each router. Its original purpose is to stop packets from looping forever — when TTL hits zero, the router that decremented it throws the packet away and, per the IP spec, sends back an ICMP “time exceeded” message to the source.

Traceroute weaponizes that behavior deliberately:

traceroute example.com

Here’s the sequence traceroute runs:

1. Send a probe with TTL = 1. The first router (your gateway) decrements it to 0, drops it, and replies “time exceeded.” That reply’s source address is hop 1.
2. Send a probe with TTL = 2. It survives the first router, dies at the second, revealing hop 2.
3. Keep incrementing TTL until a probe finally reaches the destination, which replies differently (port-unreachable or echo reply) — ending the trace.

So each line is “the router exactly N hops away told me it dropped my TTL-expired packet.” Traceroute typically sends three probes per hop, which is why you see three timing figures on each line.

 1  _gateway (192.168.1.1)   0.42 ms  0.39 ms  0.41 ms
 2  10.20.0.1 (10.20.0.1)    3.11 ms  2.98 ms  3.05 ms
 3  * * *
 4  72.14.x.x (72.14.x.x)   12.4 ms  11.9 ms  12.2 ms
 5  ...

Each number is the round-trip time (RTT) for that probe: how long it took to send the probe and receive the “time exceeded” reply back. Rising RTT as you move down the list is normal — later hops are physically and topologically farther away.

🛠️ DevOps Perspective — By default Linux traceroute sends UDP probes; you’ll often want traceroute -I (ICMP echo) or traceroute -T -p 443 (TCP SYN to a port) instead. Firewalls frequently drop UDP probes but permit the TCP port your service actually uses, so a TCP trace to port 443 more faithfully mirrors how your real traffic is treated. When a trace “dies” but the service works, switching probe type is the first thing to try.

Reading Round-Trip Time and Packet Loss

Two signals matter on each hop: latency (the RTT figures) and loss (probes that got no reply). A single traceroute gives three RTT samples per hop and a rough sense of loss — a tiny window. Before drawing conclusions, internalize three rules that trip up almost everyone.

Rule 1 — A * is not the same as failure

A * means “no reply came back for that probe within the timeout.” That can mean loss — but very often the router is perfectly healthy and simply chose not to answer. Routers treat generating ICMP “time exceeded” messages as low-priority busywork; many rate-limit or deprioritize it, and some never respond to traceroute probes at all. The router’s actual job — forwarding your real traffic — is unaffected.

The tell is what happens after the starred hop:

 6  * * *
 7  203.0.113.9   24.1 ms  23.8 ms  24.0 ms
 8  198.51.100.2  25.0 ms  24.7 ms  24.9 ms

Hop 6 shows all asterisks, but hops 7 and 8 answer normally — so traffic sailed through hop 6 just fine. Hop 6 was silent, not broken. Loss only matters when it persists from a hop all the way through to the destination.

Rule 2 — Latency at one hop that clears later isn’t that hop’s fault

You’ll sometimes see a hop with a scary RTT spike where the next hop is fast again:

 5  10.0.5.1   4.2 ms   4.1 ms   4.3 ms
 6  10.0.6.1   88.0 ms  91.2 ms  87.5 ms
 7  10.0.7.1   5.0 ms   4.9 ms   5.1 ms

If hop 6 were genuinely adding 85 ms, hop 7 (which is farther) couldn’t possibly be faster. What you’re seeing is hop 6’s control plane being slow to generate the ICMP reply to your probe, while its data plane forwards real traffic instantly. Latency that appears at one hop and disappears at the next is a property of that router’s ICMP handling, not of the path. Only latency that persists from a hop onward reflects real path delay.

Rule 3 — The return path may differ from the forward path

Traceroute measures round trips, but the internet routes each direction independently. The path your probe takes out to hop N and the path hop N’s reply takes back can be completely different networks — this is asymmetric routing, and across the internet it’s the normal case, not an anomaly. So an RTT figure blends forward and return latency, and a reverse-path problem can make a hop look bad when the forward path to it is fine. You can’t see the return path from your side at all. This is the biggest reason to treat traceroute as a hint generator, not a precise instrument.

🔎 Troubleshooting Tip — Trust trends, not single lines. One hop with high RTT or a stray * is noise. A pattern — loss or latency that begins at a specific hop and continues through every hop after it, all the way to the destination — is signal. Diagnose the boundary where the trend starts, not the individual noisy line.

MTR — Traceroute That Keeps Watching

A single traceroute is a snapshot: three probes per hop, then it’s done. Intermittent problems — the kind that drop 2% of packets or spike every few seconds — hide easily in three samples. mtr (My Traceroute) continuously re-runs the trace and accumulates statistics per hop, combining traceroute and ping into one live view.

mtr isn’t always preinstalled. On Debian/Kali:

apt-get install --no-install-recommends mtr-tiny

mtr-tiny is the terminal-only build (no GUI dependencies) — ideal for servers and containers. Once it’s installed, run it against a host:

mtr example.com

This opens a live table that updates every second:

Host                    Loss%  Snt   Last  Avg  Best  Wrst
1. _gateway              0.0%   50    0.4   0.4   0.3   0.6
2. 10.20.0.1             0.0%   50    3.0   3.1   2.9   4.2
3. 203.0.113.9           0.0%   50   12.1  12.4  11.8  40.1
4. 198.51.100.2          8.0%   50   25.3  26.0  24.1  90.4
5. example.com           8.0%   50   25.9  26.4  24.7  95.0

Now you have real statistics per hop: Loss% over many probes (far more trustworthy than three), the Snt count, and Last/Avg/Best/Wrst RTT so you can see jitter (the gap between Best and Wrst). For a scriptable one-shot report, add report mode:

mtr --report --report-cycles 100 example.com

That sends 100 cycles and prints a single summary table — perfect for pasting into a ticket or capturing before/after a change.

The same three rules still apply. In the table above, notice loss appears at hop 4 and continues at hop 5 (the destination) — that’s a real, sustained problem starting at hop 4. If instead hop 4 had shown 8% loss but hop 5 showed 0%, that would be hop 4 deprioritizing ICMP, not real loss. Loss that clears on a later hop is cosmetic; loss that persists to the end is real.

🏭 Why This Matters in Production — “The API times out sometimes” is nearly impossible to catch with a single traceroute. Leaving mtr running for a minute against the endpoint turns a vague, intermittent complaint into “consistent 8% loss beginning at our transit provider’s hop.” That evidence is what lets you open a credible ticket with the right party instead of guessing.

DevOps Perspective — Is It Me or Upstream?

The highest-value thing path analysis does is separate “my host / my network” from “somewhere upstream.” Read the trace top-down:

• Loss or failure at hop 1 (your gateway) → the problem is local: your interface, cable, wifi, or LAN router. Fix it on your side.
• Clean through your own hops, then loss begins upstream → the problem is beyond your control boundary. Escalate to your ISP, transit provider, or cloud network — with the trace as evidence.
• Clean all the way to the destination → the path is fine; the issue is higher up the stack (the application, TLS, or the port), not the network path. Move up the troubleshooting stack.

Confirm which router is your real first hop before trusting the trace, because a wrong route sends you down the wrong path entirely:

ip route get 93.184.216.34

ip route get asks the kernel exactly which source address, interface, and gateway it will use to reach that specific destination. If hop 1 in your traceroute doesn’t match the gateway ip route get reports, your routing table — not the wider internet — is the story, and you should revisit Linux routing.

🛠️ DevOps Perspective — In containers and Kubernetes the “first hop” is often a virtual bridge or overlay, not a physical router. Running ip route get <dest> inside the affected container, then tracing from there, reveals whether a broken overlay route or NAT boundary is eating traffic before it leaves the node.

Try It Yourself

🧪 Try It — Build intuition on paths you’re allowed to probe.

1. Run traceroute <your-own-server> and identify hop 1. Confirm it matches ip route get <that-server-ip>.
2. Re-run as traceroute -T -p 443 <host> and compare — does the TCP trace reach farther than the UDP one? That gap is a firewall’s probe policy.
3. Install mtr-tiny and run mtr --report --report-cycles 50 <host>. Note any hop with a lone * or RTT spike whose next hop is clean — that’s a deprioritized router, not a fault.
4. Find a hop where loss (if any) persists to the destination versus one where it clears. Only the persistent one is a real path problem.

Common Problems

Symptom	Likely meaning	Next step
* * * at one hop, later hops fine	Router rate-limits/ignores ICMP — not loss	Ignore it; read the trend past it
Trace stops entirely partway	Firewall dropping the probe type	Retry with -I (ICMP) or -T -p 443 (TCP)
RTT spike at one hop, next hop faster	Slow control plane on that router	Ignore; only sustained latency counts
Loss starts at a hop, continues to end	Real loss from that boundary onward	Localize to that hop; escalate with evidence
Hop 1 already lossy	Local network / interface / gateway	Fix locally; check cable, wifi, ip a
Trace clean but service fails	Not a path problem	Move up: port, TLS, app layer

Troubleshooting Workflow

Observe   → traceroute / mtr the destination
Hypothesis→ where does loss/latency START and PERSIST?
Test      → mtr --report for stable stats; retry -I/-T
Evidence  → sustained loss from hop N to the end
Identify  → is hop N ours, or upstream?
Correct   → fix locally, or escalate with the trace
Validate  → re-run mtr; confirm loss is gone

What You Learned

• A hop is a router on the path; traceroute discovers each one by sending probes with incrementing TTL and reading the ICMP “time exceeded” reply that router returns.
• Each line shows round-trip time from three probes — a small sample, so read trends, not single lines.
• A * usually means a router deprioritized or rate-limited ICMP, not that the hop is failing — asterisks are not failure.
• Latency or loss at one hop that clears on the next is cosmetic; only what persists to the destination is a real path problem.
• The return path can differ from the forward path (asymmetric routing), so RTT blends both directions — treat traceroute as a hint, not a precise measurement.
• mtr (install mtr-tiny) continuously accumulates per-hop loss% and latency, far better for intermittent issues than a single traceroute.
• Path analysis separates “my network” from “upstream”; confirm your real first hop with ip route get before trusting the trace.

You can now localize where on a path connectivity changes. Next we shift from mapping the path to validating what’s actually listening at the other end: Nmap for DevOps Network Validation (Part 11) — authorized port and service discovery for infrastructure checks. It pairs naturally with Testing connectivity, DNS troubleshooting, and later tcpdump packet analysis when you need to see the packets themselves.