Packet Analysis With tcpdump for DevOps

When a service “just won’t connect” and every log looks clean, the packets don’t lie. tcpdump is a command-line tool that captures the raw network traffic flowing through an interface and shows you exactly what left your host and what came back. This lesson teaches the handful of tcpdump skills a DevOps engineer actually needs to diagnose connectivity problems — no offensive techniques, just infrastructure diagnostics.

What tcpdump Does

tcpdump reads packets off a network interface and prints a one-line summary of each. That’s it — but that raw visibility is precisely what makes it invaluable when higher-level tools give you vague answers like “connection timed out” or “name resolution failed.”

Think of the layers of a failing request:

Your application says “couldn’t reach the database.”
The OS says “the connection timed out.”
tcpdump shows you the actual truth: we sent three SYN packets and never got a single reply.

That last line is the one that tells you where the problem lives — the network path, a firewall, DNS, or the remote service itself.

🛠️ DevOps Perspective — Most connectivity bugs are really questions of “did the packet leave, and did anything answer?” tcpdump answers both directly. It turns a guessing game (“maybe it’s DNS, maybe it’s the firewall”) into an observation you can point at in a postmortem.

Capturing Packets and Choosing an Interface

By default tcpdump picks an interface for you, which is rarely the one you want. Use -i to choose explicitly, and -i any to capture on every interface at once — handy when you’re not sure which one a container or route uses.

sudo tcpdump -i eth0          # capture on a specific interface
sudo tcpdump -i any           # capture across all interfaces

List the interfaces available to capture on:

sudo tcpdump -D               # -D = list capturable interfaces

Two flags make the output readable from the first packet and belong on almost every command you run:

Flag	What it does	Why you want it
`-n`	Don’t resolve IPs to hostnames	Faster, and avoids DNS lookups polluting your own capture
`-nn`	Don’t resolve IPs or port numbers	See `443` instead of `https`, `53` instead of `domain`
`-c <count>`	Stop after N packets	Keep a capture from running forever
`-t` / `-tttt`	Control timestamp format	`-tttt` prints human-readable dates

# A sensible everyday default: all interfaces, no name resolution, stop after 20 packets
sudo tcpdump -i any -nn -c 20

Press Ctrl+C to stop a running capture. tcpdump then prints a summary of how many packets it captured, received, and dropped.

🔐 Security Note — tcpdump reads raw traffic and therefore needs root (run it with sudo). A capture can contain sensitive data — credentials in plaintext protocols, session tokens, internal hostnames, and personal data. Only capture traffic on systems you own or are explicitly authorized to test, prefer your own lab or test hosts, and never capture on networks or systems that aren’t yours. Delete PCAP files when you’re done, and never paste raw captures into tickets or chat without scrubbing them first.

Filtering: Seeing Only What Matters

An unfiltered capture on a busy host is a firehose. tcpdump’s filter expressions (BPF, Berkeley Packet Filter) let you narrow the capture to exactly the conversation you care about. Filters go at the end of the command.

Filter by host

sudo tcpdump -i any -nn host 10.0.0.5          # traffic to OR from 10.0.0.5
sudo tcpdump -i any -nn src host 10.0.0.5       # only traffic FROM that host
sudo tcpdump -i any -nn dst host 10.0.0.5       # only traffic TO that host

Filter by port

sudo tcpdump -i any -nn port 443                # HTTPS traffic in either direction
sudo tcpdump -i any -nn dst port 5432           # traffic heading to Postgres

Filter by protocol

sudo tcpdump -i any -nn tcp                     # only TCP
sudo tcpdump -i any -nn udp                     # only UDP (DNS, some VPNs, etc.)
sudo tcpdump -i any -nn icmp                    # ping / unreachable messages

Combine filters

Filters compose with and, or, and not. Group with parentheses (quote them so the shell leaves them alone):

# Traffic to a specific database host, but only on the Postgres port
sudo tcpdump -i any -nn host 10.0.0.5 and tcp port 5432

# DNS traffic to either resolver
sudo tcpdump -i any -nn "udp port 53 and (host 1.1.1.1 or host 10.0.0.2)"

💡 Note — A filter that’s too broad buries the signal; one that’s too narrow can hide the very packet that proves your theory wrong. Start a little broader than you think you need (e.g. host X), confirm you see the conversation, then tighten with and port Y.

Writing and Reading PCAP Files

Printing to the terminal is fine for a quick look, but for anything you want to keep, share with a teammate, or open in a GUI like Wireshark, write the capture to a PCAP file with -w.

# Write raw packets to a file (note: -w writes binary, so no need for -n here)
sudo tcpdump -i any -w /tmp/db-timeout.pcap host 10.0.0.5 and tcp port 5432

-w stores the full packets in binary form — you won’t see anything scroll by, which is expected. Stop with Ctrl+C when you’ve captured the event.

Read a saved capture back with -r. Reading a file needs no special privileges, so you can analyze it as a normal user:

tcpdump -nn -r /tmp/db-timeout.pcap                    # replay the whole capture
tcpdump -nn -r /tmp/db-timeout.pcap tcp port 5432       # filter while reading

A common and robust workflow:

Capture broadly to a file during the failure window (-w capture.pcap).
Analyze offline afterward with -r and different filters, as many times as you like.
Hand the same PCAP to a teammate or open it in Wireshark for deeper inspection.

🛠️ DevOps Perspective — Capturing to a file decouples observing the incident from analyzing it. During an outage you rarely have time to craft the perfect filter — capture the window to disk, restore the service, and dissect the PCAP once the pressure is off.

Reading the Output: TCP Flags and What They Mean

The one-line summaries carry the diagnosis if you know what to look for. For TCP, the flags in each line tell the story of the connection handshake:

Flag in output	Meaning	What it tells you
`[S]`	SYN	A connection attempt starting
`[S.]`	SYN-ACK	The other side accepted and is responding
`[.]`	ACK	Normal acknowledgement — data flowing
`[P.]`	PSH-ACK	Data being pushed to the application
`[F.]`	FIN	A graceful connection close
`[R]` / `[R.]`	RST	Connection reset — actively refused or torn down

A healthy TCP connection opens with the three-way handshake — SYN → SYN-ACK → ACK:

IP 10.0.0.10.51514 > 10.0.0.5.5432: Flags [S]   # we send SYN
IP 10.0.0.5.5432 > 10.0.0.10.51514: Flags [S.]  # they answer SYN-ACK
IP 10.0.0.10.51514 > 10.0.0.5.5432: Flags [.]   # we ACK — connection is up

If you see the SYN go out but nothing comes back, or a RST comes back instead, you’ve found your problem — the sections below show exactly those patterns.

DevOps Diagnostics: Common Failures in Packets

Here’s how the everyday connectivity failures look through tcpdump.

DNS failure

Symptom: an app reports “could not resolve host” or hangs before any connection attempt. Watch DNS (UDP port 53):

sudo tcpdump -i any -nn port 53

IP 10.0.0.10.40311 > 10.0.0.2.53: A? api.internal.example. (36)   # query goes out
# ...and no response line ever appears

If you see the query leave but never see a reply, name resolution is broken upstream — the resolver is unreachable or not answering. If you see a reply with NXDomain, the name genuinely doesn’t exist. Either way, the problem is DNS, not your application.

Connection timeout

Symptom: “connection timed out.” The classic signature is repeated SYNs with no answer:

IP 10.0.0.10.51600 > 10.0.0.5.5432: Flags [S]   # SYN
IP 10.0.0.10.51600 > 10.0.0.5.5432: Flags [S]   # SYN again (retransmit)
IP 10.0.0.10.51600 > 10.0.0.5.5432: Flags [S]   # SYN again — still no reply

Three lonely SYNs and no SYN-ACK means the packet is going into a black hole — typically a firewall or security group silently dropping the traffic, or the host being down. The traffic left your box, so your local config is probably fine; look at the path and the far end.

TCP reset (RST) — connection refused

Symptom: “connection refused,” and it fails instantly rather than hanging:

IP 10.0.0.10.51700 > 10.0.0.5.5432: Flags [S]    # SYN
IP 10.0.0.5.5432 > 10.0.0.10.51700: Flags [R.]   # RST — actively refused

A RST in response to your SYN means the host is reachable but nothing is listening on that port (or a firewall is configured to reject rather than drop). The service isn’t running, is bound to the wrong interface, or is on a different port than you think.

Failed application connection

Sometimes the TCP handshake succeeds but the application layer still fails. If you see the full SYN / SYN-ACK / ACK, then a quick RST or FIN right after data is exchanged, the network is fine — the failure is in the application (auth rejection, protocol mismatch, TLS handshake failure). That tells you to stop debugging the network and start reading application and TLS logs. (For the TLS side of this, see TLS Certificate Troubleshooting.)

Unexpected network path

Symptom: traffic works from one host but not another, or hits the wrong destination. Watch where packets actually go:

sudo tcpdump -i any -nn host api.internal.example

If the destination IP in the capture isn’t the one you expect — a stale DNS record, a wrong /etc/hosts entry, or a NAT rule sending you somewhere unintended — you’ll see it immediately in the > destination of each line. The packets reveal the real path, not the one your config claims.

🔎 Troubleshooting Tip — The two fastest diagnoses in TCP: a SYN with no SYN-ACK means the traffic is being filtered or dropped (firewall / security group / down host) — it fails slowly with a timeout. A RST means the connection was actively refused (nothing listening, or a reject rule) — it fails instantly. Timeout vs. instant failure is your first clue before you even open a capture.

Try It: Watch DNS Resolution Live

Do this on a lab host or a VM you control (see Your First DevOps Security Lab).

🧪 Try It — Open two terminals on your lab host. In the first, start capturing DNS traffic:
sudo tcpdump -i any -n port 53
In the second terminal, trigger a lookup against a public resolver:
dig @1.1.1.1 example.com
Back in the first terminal you’ll see the query leave and the response return — something like A? example.com. going out and an answer coming back. Now try a name that doesn’t exist (dig @1.1.1.1 thisdoesnotexist.example) and watch for the NXDomain response. You’ve just watched name resolution succeed and fail at the packet level — the exact skill you’ll use when an app “can’t resolve” a host in production.

When to Reach for tcpdump (and When Not To)

tcpdump is the right tool when:

A connection fails and higher-level tools only say “timeout” or “refused” without saying why.
You need to confirm whether traffic is even leaving a host, or where it’s actually going.
You suspect DNS, a firewall, a security group, or a wrong route, and want proof rather than a hunch.
You want to hand a reproducible capture (a PCAP) to a teammate or attach it to an incident.

It’s overkill when a simpler check answers the question — dig for DNS (DNS Troubleshooting), curl for HTTP, or ss -tlnp to see what’s listening locally. Reach for packets when the simpler tools disagree with reality.

⛔ Production Warning — Running a broad capture on a busy production host adds load and records potentially sensitive traffic to disk. If you must capture in production, scope it tightly with host/port filters, cap it with -c or a short time window, write to a secure location, and delete the PCAP as soon as you’ve analyzed it.

This lesson builds directly on the concepts in Networking Fundamentals — if terms like interface, port, or the TCP handshake feel shaky, that’s the lesson to revisit first.

What You Learned

tcpdump captures raw packets off an interface, giving you ground truth when logs and error messages are vague — always run it with sudo because it needs root.
-i selects the interface (-i any for all), -nn keeps output readable, and -c caps the capture so it doesn’t run forever.
BPF filters narrow the firehose — combine host, port, and tcp/udp/icmp with and/or/not to isolate a single conversation.
-w writes a PCAP and -r reads it back, letting you capture during an incident and analyze offline (or in Wireshark) afterward.
TCP flags tell the diagnosis: a SYN with no SYN-ACK = filtered or dropped (slow timeout); a RST = actively refused (instant failure); a missing DNS reply on port 53 = broken name resolution.
Captures contain sensitive data — only capture traffic you own or are authorized to test, keep filters tight, and delete PCAP files when you’re done.