Build a Complete DevOps Network Troubleshooting Lab

You have spent fourteen lessons learning individual tools: ip addr, ip route, dig, ss, nc, curl, nmap, tcpdump, and the firewall utilities. Individually, each is a party trick. Together, in the hands of someone who knows which one to reach for, they are a complete diagnostic method. This capstone is where the tools stop being a checklist and become a workflow.

We are going to build one small, self-contained lab — a Kali container, an nginx web server, and a tiny API service on an isolated Docker network — and then deliberately break it six different ways. Each break is a real failure mode you will meet in production: a service bound to loopback, a bad hostname, a wrong port, a container on the wrong network, a firewall dropping traffic, and a TLS certificate that does not match its host. For each one you will observe a symptom, gather evidence with the right tool, compare expected against observed, and only then apply a fix. That last discipline is the whole point: anyone can restart a container, but a real infrastructure engineer names the broken layer first.

What You Will Learn

How to stand up an isolated, reusable network lab with docker compose
Which tool answers which question, and in what order to reach for them
The Expected → Evidence → Observed → Compare → Identify → Fix → Validate loop, applied to six distinct faults
How to tell a loopback-bind from a wrong-port from a wrong-network problem without guessing
How to read the difference between a dropped and a rejected packet, and a hostname mismatch in a certificate

The Lab Architecture

Everything lives on one isolated Docker bridge network. Nothing is exposed to your real LAN, so you can break things freely.

             DevOps Networking Lab
                   Kali
                     |
              Lab Network (isolated bridge)
                     |
      +--------------+--------------+
      |                             |
  Web Server                     API Server
    (nginx)                      (service)

Kali is your workstation inside the network — it sits on the same bridge as the two services, so every probe you run is a probe from a client that should legitimately be able to reach them. When a probe fails, that failure is real signal, not an artifact of Docker’s port publishing.

Here is the project layout you will build:

kali-network-lab/
├── compose.yaml
├── kali/
│   └── Dockerfile
├── nginx/
│   └── default.conf
├── api/
│   └── (small http listener)
└── captures/        # tcpdump output lands here

🛠️ DevOps Perspective — This is the same shape as a real service mesh in miniature: a client, a frontend, a backend, and a network they share. Every skill you practice here — “is it listening, is it resolvable, is it routable, is it filtered” — is exactly what you do when a pod cannot reach a service in Kubernetes or an app cannot reach its database in a VPC. The topology is small; the method is production-grade.

Building the Lab

compose.yaml

We use modern docker compose with a compose.yaml file. The critical piece is networks: — we define our own user-defined bridge so the containers get Docker’s embedded DNS and predictable addressing, isolated from everything else on the host.

name: kali-network-lab

networks:
  labnet:
    driver: bridge

services:
  kali:
    build: ./kali
    container_name: lab-kali
    # NET_RAW lets tcpdump/nmap craft and read raw packets.
    # We add ONLY this capability — never --privileged.
    cap_add:
      - NET_RAW
    networks:
      - labnet
    volumes:
      - ./captures:/captures
    command: sleep infinity

  web:
    image: nginx:stable
    container_name: lab-web
    networks:
      - labnet
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro

  api:
    image: python:3.12-slim
    container_name: lab-api
    networks:
      - labnet
    working_dir: /app
    volumes:
      - ./api:/app:ro
    command: python3 -m http.server 80

Read what this declares before you run it. The kali service builds from our own Dockerfile and gets cap_add: [NET_RAW] — that single capability is what lets tcpdump and nmap open raw sockets. We do not use --privileged, which would hand the container near-total access to the host kernel, and we never mount /var/run/docker.sock, which would let a container control the Docker daemon itself. Grant the narrowest capability that does the job and no more. The web and api services need no special capabilities — they only listen.

🔐 Security Note — Everything in this lab is a system you own, running on your own machine, on an isolated network. Only scan, inspect, or test systems you own or have explicit permission to assess. This is authorized infrastructure validation and defensive troubleshooting, not hacking. Keep every probe you learn here pointed at reader-owned infrastructure and controlled labs.

kali/Dockerfile

The Kali container needs the toolkit installed. This Dockerfile follows the standard hygiene rules: --no-install-recommends to avoid pulling in a desktop’s worth of extras, and a cleanup step in the same layer so the apt cache does not bloat the image.

FROM kalilinux/kali-rolling

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        iproute2 \
        dnsutils \
        iputils-ping \
        netcat-traditional \
        curl \
        openssl \
        traceroute \
        nmap \
        tcpdump \
        ca-certificates && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /root
CMD ["sleep", "infinity"]

Every package here maps to a tool you already know: iproute2 gives you ip and ss, dnsutils gives you dig, netcat-traditional gives you nc, and the rest are named for what they are. Combining apt-get clean && rm -rf /var/lib/apt/lists/* in the same RUN layer keeps the image small — if you cleaned in a later layer the cache would still be baked into an earlier one.

The support files

The nginx/default.conf and the api/ directory hold the service config we will bend in each scenario. For now, start with a healthy baseline: nginx serving on port 80 for all addresses, and the API as a plain python3 -m http.server. Bring the lab up:

cd kali-network-lab
docker compose up -d --build
docker compose ps

docker compose ps should show lab-kali, lab-web, and lab-api all running. From here on, “open a shell in Kali” means:

docker compose exec kali bash

Every diagnostic command below runs inside that Kali shell, as a client on labnet, exactly like a real workstation on the service network.

The Diagnostic Loop

Before the first break, fix the method in your head. Every scenario runs the same loop:

Expected  → what SHOULD happen
Evidence  → run the RIGHT tool to observe reality
Observed  → what ACTUALLY happens
Compare   → where do expected and observed diverge?
Identify  → which layer is broken?
Fix       → correct THAT layer only
Validate  → re-run the original test; confirm green

Map every fault onto the troubleshooting stack you learned in the networking fundamentals lesson:

Application → TLS → Port → DNS → Gateway → Route → Interface

The skill is not knowing the fixes — it is knowing which layer a symptom points to, so you gather the right evidence instead of guessing. Now let us break things.

Scenario 1 — The Service Only Answers Itself

Symptom. From Kali, curl http://lab-web/ hangs and then fails with connection refused, even though docker compose ps swears lab-web is up and healthy.

Which tools: curl and nc to confirm the failure (testing connectivity), then ss on the web container to see the bind address.

Expected. A web server should accept connections from any client on its network — its socket should be bound to 0.0.0.0:80 (all interfaces).

Gather evidence. First reproduce the failure from Kali, then look at what the web container is actually listening on:

# From Kali
curl -v --max-time 5 http://lab-web/

# From the web container
docker compose exec web ss -tulpn

Observed. The curl fails fast with “connection refused” — an instant failure, not a hang, which the TCP vs UDP lesson taught you means the host is up and actively rejecting, not dropping. And ss shows the listener as 127.0.0.1:80 rather than 0.0.0.0:80.

Compare and identify. Expected 0.0.0.0:80; observed 127.0.0.1:80. The service is bound to loopback only, so it answers requests from inside its own container and refuses everyone else. This is the single most common “it works on my machine” fault, and it lives at the Port/bind layer — the service is running, it is just not reachable. DNS, routing, and firewalls are not involved.

Fix. Reconfigure the service to listen on 0.0.0.0 (for nginx, listen 0.0.0.0:80; in default.conf; for the Python listener, python3 -m http.server --bind 0.0.0.0 80), then recreate the container.

Validate. Re-run curl http://lab-web/ from Kali — you should now get a 200. Confirm the fix at the socket layer with ss -tulpn showing 0.0.0.0:80. See ports and listening services for a deeper drill on the loopback-vs-all-interfaces distinction.

Scenario 2 — The Name That Points Nowhere

Symptom. The API’s config tells it to call the web tier at http://webserver/, and every request dies immediately with “could not resolve host.”

Which tools: dig against Docker’s embedded DNS, and curl to confirm the error class.

Expected. On a user-defined bridge, Docker runs an embedded DNS resolver at 127.0.0.11 that resolves every service name to its container IP. The service is named web in compose.yaml, so dig web should return an address.

Gather evidence.

# From Kali — the name the app was told to use
dig +short webserver

# The name that actually exists
dig +short web

Observed. dig webserver returns nothing (NXDOMAIN — an empty answer), while dig web returns a 172.x address on labnet. And any curl http://webserver/ fails with “could not resolve host,” the classic DNS/name-resolution signature.

Compare and identify. Expected the app’s target name to resolve; observed that webserver does not exist but web does. This is not a network fault at all — the packets never leave, because name resolution fails before a connection is ever attempted. The broken layer is DNS. Docker’s embedded DNS only knows the service names you declared; webserver was never one of them.

Fix. Point the app at the real service name (web), or add a network alias in compose.yaml (networks: labnet: aliases: [webserver]) so the intended name resolves.

Validate. dig +short webserver now returns the container IP, and the app’s request succeeds. The DNS troubleshooting lesson covers embedded-DNS quirks and search domains in depth.

🔎 Troubleshooting Tip — “Could not resolve host” and “connection refused” look similar in a panic but point at opposite layers. Resolution failure means you never got an IP (fix DNS). Refused means you got an IP, reached the host, and it said no (fix the port/service). Always read the exact error string before you touch anything.

Scenario 3 — Right Service, Wrong Port

Symptom. The API listener is up, but curl http://lab-api/ from Kali fails with connection refused. The team insists “the API is running — I can see the process.”

Which tools: nmap to discover which ports actually answer, then ss and curl to confirm.

Expected. The client is calling port 80. If the service listens on 80, a connect should succeed.

Gather evidence. Instead of assuming the port, ask the host which ports are open:

# From Kali — scan the common range on the API host
nmap -p 1-1000 lab-api

# Confirm on the API host itself
docker compose exec api ss -tulpn

Observed. nmap reports port 80 as closed but port 8080 as open. ss on the API container confirms the listener is on :8080. A direct curl http://lab-api:8080/ returns 200, while curl http://lab-api/ (port 80) is refused.

Compare and identify. Expected port 80 open; observed 80 closed and 8080 open. The service is perfectly healthy — it is simply listening on a different port than the client is dialing. This is a Port layer mismatch, a configuration disagreement between two ends. nmap earned its keep here: it told you where the service actually is instead of only confirming that your chosen port was closed.

Fix. Align the two ends — either configure the service to listen on 80, or update the client/config to call 8080. In compose.yaml you would set the command to python3 -m http.server 80.

Validate. nmap -p 80 lab-api reports open, and curl http://lab-api/ returns 200. Revisit Nmap for network validation for reading open vs closed vs filtered.

Scenario 4 — On the Wrong Network Entirely

Symptom. A new cache service was added, but from Kali it is completely unreachable — ping lab-cache cannot even resolve or route to it, and curl times out with “no route to host.”

Which tools: docker network inspect, then ip route and ping from inside Kali.

Expected. Every service that Kali needs to reach must be attached to labnet. Containers on the same user-defined bridge share a subnet and can route to each other directly.

Gather evidence. Ask Docker which containers are actually on the network, then check reachability:

# On the host — who is really attached to labnet?
docker network inspect kali-network-lab_labnet \
  --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'

# From Kali — do we even have a route to its subnet?
ip route
ping -c 2 lab-cache

Observed. docker network inspect lists lab-kali, lab-web, and lab-api on labnet — but not lab-cache. From Kali, ip route shows a route only for labnet’s subnet, and ping lab-cache fails to resolve or returns “no route to host” because the cache sits on a different bridge with a different subnet.

Compare and identify. Expected all four services on labnet; observed the cache missing from it. The service is running fine, but it is isolated on the wrong network, so there is no shared subnet and no route between it and Kali. This is a Route/Interface layer fault — the “no route to host” signature you learned to read points straight at network topology, not at the service.

Fix. Attach the container to labnet in compose.yaml (add labnet under its networks:), or connect it live with docker network connect kali-network-lab_labnet lab-cache.

Validate. docker network inspect now lists the cache with a labnet address, ip route in Kali covers its subnet, and ping lab-cache succeeds. The Linux routing lesson explains why a missing route produces exactly this signature.

🏭 Why This Matters in Production — “Wrong network” is the container-era version of a VLAN or VPC-subnet mistake. In Kubernetes it shows up as a pod that cannot reach a Service because a NetworkPolicy sits between them. The tool changes — docker network inspect becomes kubectl get networkpolicy — but the question is identical: are these two things actually on a path that can carry a packet between them?

Scenario 5 — Dropped, Not Refused

Symptom. curl http://lab-web/ from Kali no longer fails fast — it hangs for the full timeout and then reports a timeout. Yesterday it was instant. Nothing about the web service changed.

Which tools: nmap to read filtered vs closed, and curl/nc to feel the difference between timeout and refused.

Expected. A reachable port returns quickly — either open (handshake completes) or closed (instant RST). A slow hang is a different animal.

Gather evidence. Compare the speed and shape of the failure, then let nmap classify the port:

# From Kali — note whether this is instant or a slow hang
nc -vz -w 5 lab-web 80

# nmap distinguishes filtered (dropped) from closed (rejected)
nmap -p 80 lab-web

Observed. nc hangs for the full 5-second window, then fails — a timeout, not an instant “refused.” nmap reports port 80 as filtered, meaning it got no response at all: the SYN went out and nothing came back.

Compare and identify. Expected a fast open or a fast closed; observed a slow filtered. That distinction is the entire diagnosis. A rejected packet (RST) comes back instantly and reports “closed/refused” — the host is up and saying no. A dropped packet gets silence; the sender waits until it gives up, producing a timeout and nmap’s filtered. Silence means a firewall with a DROP rule is swallowing the traffic. The broken layer is a firewall/filter in the path, not the service.

If you want ground truth, tcpdump -ni any tcp port 80 on the path shows your SYN going out with no reply ever returning — the visual signature of a drop. The tcpdump packet analysis lesson covers reading that silence.

Fix. Inspect the firewall on the path (iptables -L -n / nft list ruleset) and change the offending rule — or, if the policy should REJECT rather than DROP, at least make failures fast and honest. The service itself needs no change.

Validate. After correcting the rule, nmap -p 80 lab-web reports open and nc -vz lab-web 80 succeeds instantly. The Linux firewall troubleshooting lesson covers the DROP-vs-REJECT decision and how each one feels to a client.

🔎 Troubleshooting Tip — The speed of a failure is free diagnostic data. Instant failure = you reached a host that said no (refused/closed → check the service). Slow failure that ends in a timeout = your packet vanished (dropped/filtered → check firewalls, routes, security groups). You can often name the layer before you have run a single tool, just from how long the failure took.

Scenario 6 — The Certificate That Lies About Its Name

Symptom. After enabling HTTPS on the web tier, curl https://lab-web/ fails with a certificate error, even though the TLS handshake itself completes and the page loads over plain HTTP.

Which tools: openssl s_client to read the certificate’s identity, and curl -v to see the verification error.

Expected. A TLS certificate presented by lab-web must list lab-web in its Common Name (CN) or Subject Alternative Name (SAN). The client verifies that the name it dialed matches a name the certificate claims.

Gather evidence. Ask the server to show its certificate and read the names on it:

# From Kali — pull the cert and read CN + SANs
echo | openssl s_client -connect lab-web:443 -servername lab-web 2>/dev/null \
  | openssl x509 -noout -subject -ext subjectAltName

# And see curl's exact verification complaint
curl -v https://lab-web/

Observed. The certificate’s subject is CN=example.local with a SAN of DNS:example.local — but you connected to lab-web. curl -v reports something like “certificate subject name does not match target host name.” The TCP connection and TLS negotiation succeeded; verification is what failed.

Compare and identify. Expected a certificate naming lab-web; observed one naming example.local. The transport is healthy — port open, handshake done — so the fault sits at the TLS layer, specifically identity verification. The cert is real and valid; it is simply for a different name than the one the client used. This is the most common TLS failure in service-to-service traffic: a certificate minted for one hostname served on another.

Fix. Reissue the certificate with a SAN that includes lab-web (SANs are what modern clients check; CN alone is deprecated), or have the client connect using the exact hostname on the certificate.

Validate. Re-run the openssl s_client command — the SAN should now list DNS:lab-web — and curl https://lab-web/ completes with no certificate warning. For the mechanics of reading certs and chains, see Kali on Docker: TLS certificate troubleshooting.

The Master Toolkit You Now Wield

Six faults, six layers, and a single tool (or a small pair) decisive for each. This is the kit you now carry, and the question each one answers:

ip addr      — which interfaces and IPs do I have?
ip route     — where do packets for this destination go?
ip neigh     — who is on my local link? (ARP/neighbor table)
dig          — does this name resolve, and to what?
ping         — is the host reachable at all?
ss           — what is listening here, and on which address?
nc           — can I open a TCP/UDP connection to this port?
curl         — does the application actually respond?
openssl      — is the TLS certificate valid and correctly named?
traceroute   — what path do packets take, and where do they stop?
nmap         — which ports are open / closed / filtered?
tcpdump      — what is really on the wire, packet by packet?

The mastery is not the list — it is the mapping. A resolution error sends you to dig; an instant refusal to ss; a slow timeout to nmap and then the firewall; a cert warning to openssl. You stopped reaching for tools at random the moment you started reading symptoms as layers.

Try It Yourself

🧪 Try It — Break the lab yourself, blind. Ask a colleague (or a second terminal and your future self) to introduce one of the six faults into compose.yaml or a config file without telling you which. Bring the lab up, then diagnose it using only the loop: reproduce the symptom, read the shape of the failure, pick the one tool the symptom points to, and name the layer before you open the config to confirm. Then fix exactly that layer and validate. Do all six. When you can name the broken layer from the symptom alone — before running a tool — you have internalized the method.

Common Problems

You fixed the wrong layer. The classic trap: a DNS failure gets “fixed” by restarting the service, which of course changes nothing. Read the exact error string first — “could not resolve” is never a service problem.
You trusted docker compose ps over the socket. “Up” means the container is running, not that anything useful is listening on the address and port you expect. ss -tulpn is the truth; ps is a rumor.
You conflated refused with timeout. Instant refusal (RST) and slow timeout (drop) point at opposite layers. Time your failures; do not lump them together.
You forgot the client’s perspective. A service can be perfectly healthy and still unreachable because of DNS, ports, networks, or firewalls between you and it. Always test from the client that is actually failing — which is exactly why Kali lives on labnet.

What You Learned

How to build an isolated, reusable network lab with docker compose, a user-defined bridge, and a Kali toolbox container — using cap_add: [NET_RAW] instead of --privileged and never mounting the Docker socket.
Six real failure modes and the layer each lives at — loopback bind (Port), unresolvable name (DNS), wrong port (Port), wrong network (Route), dropped traffic (Firewall), and cert-name mismatch (TLS).
Which tool answers which question — ss for binds, dig for names, nmap for ports and filtering, docker network inspect + ip route for topology, openssl for certificates.
The Expected → Evidence → Observed → Compare → Identify → Fix → Validate loop, applied until naming the broken layer became reflex rather than guesswork.
That the speed and text of a failure are diagnostic data — instant refused vs slow timeout, “could not resolve” vs “no route to host” — each one routes you to the correct layer before you run a second command.

You Can Now Troubleshoot the Network Stack

Step back and look at what you can now investigate on any Linux host, container, or cloud instance:

Interfaces, IP addresses, and subnets — what the host is and where it sits (ip addr, CIDR math)
Routes and gateways — where its packets go and how they leave (ip route)
ARP and neighbors — who shares its local link (ip neigh)
TCP and UDP behavior — handshakes, resets, and why one hangs while the other refuses
Ports and listening services — what is actually accepting connections, and on which address (ss)
DNS resolution — whether a name resolves and to what (dig)
Network paths — where traffic travels and where it stalls (traceroute, mtr)
Firewalls — dropped vs rejected, filtered vs closed
Packet captures — the ground truth on the wire (tcpdump)
Application connectivity end-to-end — from name, to route, to port, to TLS, to a working 200 (curl, openssl)

That is the full stack, from a cable to a 200 OK. You are no longer reaching for tools at random; you read a symptom, name the layer, and pick the one instrument that settles it.

Continue Your Kali Linux Journey

You have finished the networking series. To keep building:

Return to the Kali Linux hub for the full learning path and the other series.
Work through the Kali Linux on Docker series, which takes these same skills deeper into containers — user-defined networks, embedded DNS, in-container packet capture, and TLS troubleshooting.
Sharpen your container fundamentals in the Docker Academy.

Coming next: Kali Linux + Kubernetes. The natural continuation takes the exact method you just practiced — is it listening, is it resolvable, is it routable, is it filtered — and applies it to pods, Services, DNS, and NetworkPolicies in a cluster. When a pod cannot reach a Service, you already know the questions to ask; that path will teach you the cluster-native tools that answer them. It is in production and lands soon — watch the hub for it.

You built the lab, broke it six ways, and diagnosed every fault by naming the layer before touching a fix. That is the difference between someone who restarts containers and hopes, and an engineer who troubleshoots the network stack with intent.