Linux Network Troubleshooting: Firewalls and Filtering |

You have confirmed the route exists, the neighbor resolves, and the service is listening — and yet the connection still fails. When everything at the lower layers looks healthy but traffic still will not flow, the next suspect is a firewall: a set of rules that silently decides which packets live and which packets die. This is one of the most frustrating classes of network problem because a firewall does not have to announce itself. It can simply drop a packet and let you sit there watching a connection time out with no explanation at all.

This lesson teaches you to read Linux firewall rules with nft and iptables and, more importantly, to reason about the layered filtering that surrounds any real service. A connection can be blocked in a cloud security group, a host firewall, a container network policy, or the application itself, and each one leaves a different fingerprint. Your job is not to flush rules and hope; it is to locate which layer is blocking before you touch anything.

What You Will Learn

What packet filtering is and how the INPUT, OUTPUT, and FORWARD chains differ
Why a stateful firewall lets return traffic back in via connection tracking
How to read a modern ruleset with nft list ruleset and a legacy one with iptables -L -n -v
How nftables replaced iptables and why iptables is now usually a compatibility shim
The layered filtering model — cloud, host, network ACL, Kubernetes, container — and how to tell them apart
How DROP versus REJECT tells you which layer is blocking you
A disciplined workflow that locates the blocking layer instead of blindly flushing rules

What a Packet Filter Actually Does

A firewall on Linux is a packet filter. As each packet passes through the kernel’s networking stack, the firewall compares it against an ordered list of rules and takes an action — typically accept (let it through), drop (discard silently), or reject (discard and send back an error). Rules match on source and destination IP, protocol (TCP, UDP, ICMP), and port.

Filtering happens at specific hook points in the packet’s journey, and those hooks are organized into chains. Three chains matter most for troubleshooting:

        ┌─────────────────────────────┐
        │        Linux kernel         │
        │                             │
in ───▶ │  INPUT   (to this host)     │
        │  FORWARD (through this host) │ ───▶ out
        │  OUTPUT  (from this host)   │
        └─────────────────────────────┘

INPUT filters packets destined for a process on this host. If you cannot reach a service running locally, INPUT is a prime suspect.
OUTPUT filters packets generated by this host. If this box cannot reach the outside world, check OUTPUT.
FORWARD filters packets passing through this host to somewhere else. This is the chain that matters when the machine is a router, a NAT gateway, or a container host bridging traffic between networks.

Knowing which chain governs your traffic is the first cut in any firewall diagnosis. A blocked inbound request to your web server is an INPUT question. A container that cannot reach the internet is usually a FORWARD (and NAT) question on the Docker host, not INPUT.

🛠️ DevOps Perspective — On a Kubernetes node or a Docker host, most of the interesting rules live in FORWARD, not INPUT. The node is routing packets between the pod network and the outside world, so it is forwarding, not receiving. If you go straight to INPUT you will miss the rule that is actually dropping your pod-to-pod or pod-to-internet traffic.

Stateful Firewalls and Connection Tracking

Early packet filters were stateless: every packet was judged on its own, with no memory of what came before. That is painful, because a normal TCP connection has packets flowing both directions — if you allow an outbound request but not its reply, the reply gets dropped and the connection hangs.

Modern Linux firewalls are stateful. They use conntrack (connection tracking) to remember every connection the host has seen and classify each packet by state:

NEW — the first packet of a connection you have not seen before.
ESTABLISHED — a packet belonging to a connection already in progress.
RELATED — a packet starting a new connection that is expected because of an existing one (for example, an FTP data channel, or an ICMP error about an existing flow).
INVALID — a packet that does not match any known connection state.

This is why almost every real ruleset begins with a rule like “accept everything ESTABLISHED or RELATED.” It means: once I allow a connection, let its return traffic back in automatically. You then only need explicit rules for NEW inbound connections — the ones you actually want to expose.

Client                         Server firewall
  │  NEW  (SYN) ──────────────▶ │  matched by an explicit
  │                             │  "accept NEW to :443" rule
  │  ◀──────── ESTABLISHED     │  matched by the blanket
  │           (SYN-ACK)         │  "accept ESTABLISHED" rule

🔎 Troubleshooting Tip — If new connections succeed but long-lived ones mysteriously die after a while, suspect conntrack. A full connection-tracking table (nf_conntrack: table full, dropping packet in dmesg) or an aggressive timeout will start dropping ESTABLISHED packets, and it looks exactly like a flaky network.

Reading a Modern Ruleset with nftables

Most current distributions — recent Debian, Ubuntu, RHEL, Fedora — use nftables as the underlying firewall framework. Its command-line tool is nft, and the single most useful command for troubleshooting is:

nft list ruleset

This prints the entire active firewall configuration: every table, every chain, and every rule, in nftables’ own syntax. A trimmed example looks like this:

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif "lo" accept
    tcp dport 22 accept
    tcp dport 443 accept
  }
  chain forward {
    type filter hook forward priority 0; policy drop;
  }
  chain output {
    type filter hook output priority 0; policy accept;
  }
}

Read each chain from the top down, because rules are evaluated in order and the first match wins. Here the input chain has a policy of drop — any packet reaching the end without matching a rule is discarded. The first rule accepts established,related (stateful return traffic), then loopback, then new connections to TCP 22 (SSH) and 443 (HTTPS). Anything else — say port 5432 (PostgreSQL) — matches nothing, falls through, and hits the drop policy. That is why your database connection times out: it was silently dropped by the default policy, not by an explicit rule you can grep for.

Notice the forward chain policy is also drop. On a plain host that is fine. On a router or container host it means nothing is forwarded unless a rule allows it — a common cause of “my containers have no internet.”

Reading a Legacy Ruleset with iptables

You will still meet iptables constantly, because it was the standard for two decades and countless scripts, tutorials, and tools are written against it. The workhorse command is:

iptables -L -n -v

Each flag earns its place:

-L lists the rules in all chains.
-n prints addresses and ports numerically — no reverse-DNS lookups. Without -n, iptables tries to resolve every IP to a hostname, which is slow and, ironically, can hang for a long time on the very host whose networking you are debugging. Always use -n.
-v is verbose, and it adds the two columns that matter most: pkts and bytes counters for each rule.

Chain INPUT (policy DROP 12 packets)
 pkts bytes target  prot  source       destination
 8422  1M   ACCEPT  all   0.0.0.0/0    0.0.0.0/0    ctstate RELATED,ESTABLISHED
  340 20K   ACCEPT  tcp   0.0.0.0/0    0.0.0.0/0    tcp dpt:22
    0    0   ACCEPT  tcp   0.0.0.0/0    0.0.0.0/0    tcp dpt:443

Those counters are your best friend: they tell you whether a rule is actually being hit. In the example above, the port 443 rule shows 0 packets and 0 bytes — no HTTPS traffic has ever matched it. If a user swears they are hitting your HTTPS endpoint and this counter stays at zero, their packets are not even reaching this rule — they are being blocked upstream (a cloud security group, a load balancer, an earlier DROP rule). If instead the counter climbs but the connection still fails, the packet is arriving and being handled here — so this host is where you look. Counters turn “I think the firewall is blocking it” into evidence.

🔐 Security Note — Modify firewall rules only on systems you own or administer, and think before every change. A single wrong rule — most classically, dropping INPUT before you have allowed SSH — can lock you out of a remote machine with no way back in short of console access. On production hosts, stage changes behind a timed rollback or a second session you keep open, and never test firewall edits on infrastructure you do not control.

nftables vs iptables: What You Are Really Looking At

Here is the subtlety that trips people up. On most modern systems, when you run iptables you are not talking to the old kernel subsystem at all — you are talking to iptables-nft, a compatibility shim that translates classic iptables syntax into nftables rules under the hood. Both iptables -L and nft list ruleset can show views of the same underlying rules in different syntax.

So during troubleshooting, be willing to look at both. A rule added by Docker, Kubernetes, firewalld, or ufw may be clearer in one view than the other. If iptables -L -n -v looks suspiciously empty on a machine you know has a firewall, run nft list ruleset — the rules are probably there, expressed natively in nftables. The frameworks (firewalld, ufw) are just management front-ends; underneath, it is all nftables now.

The Layered Filtering Model

The single most important mental shift in firewall troubleshooting is this: on any real deployment there is not one firewall — there are several, stacked in a line. A packet has to survive every layer to reach the application, and it can be dropped at any of them.

Client
  │
  ▼
Cloud Firewall / Security Group
  │
  ▼
Host Firewall  (nftables / iptables)
  │
  ▼
Container / K8s NetworkPolicy
  │
  ▼
Application

Walk the layers, because each one belongs to a different system and often a different team:

Cloud security groups — Virtual, stateful firewalls attached to instances in AWS, GCP, Azure, and others. They live outside your VM, so nothing inside the guest — not nft, not iptables — can show them. A security group that does not allow your port drops the packet before it ever touches the host kernel. This is the number-one cause of “the host firewall is wide open but I still time out.”
Network ACLs — Subnet-level, often stateless rules in cloud VPCs. Being stateless, they can allow the inbound request but silently drop the reply if the ephemeral return-port range is not also allowed — producing baffling half-working connections.
Host firewall — The nftables/iptables rules inside the VM, the ones this lesson has been reading. The layer you can inspect directly.
Kubernetes NetworkPolicies — Pod-level rules enforced by your CNI (Calico, Cilium, and so on). By default a pod accepts traffic from anywhere; the moment a NetworkPolicy selects a pod, everything not explicitly allowed is denied — a classic cause of pods that suddenly cannot talk to each other. Inspect them with kubectl get networkpolicy, not nft.
OpenStack security groups — The OpenStack equivalent, implemented at the hypervisor/virtual-switch layer via Neutron. Same principle: a rule outside your instance can drop traffic before the guest sees it, and you manage it through OpenStack, not the guest OS. See the OpenStack troubleshooting hub and the OpenStack category.

🏭 Why This Matters in Production — When a service is unreachable, the instinct is to log into the host and inspect its firewall. But in a cloud or Kubernetes deployment, the blocking layer is frequently not on the host at all — it is a security group, a network ACL, or a NetworkPolicy owned by the networking or platform team. Spending an hour reading host rules for a problem that lives in a security group is a very common, very avoidable waste. Rule the outer layers in or out early.

DROP vs REJECT: Reading the Fingerprint

Here is where troubleshooting becomes deduction. When a firewall blocks a packet it can do so in two fundamentally different ways, and the symptom you observe tells you a great deal about which layer did it.

DROP — The packet is discarded silently. Nothing is sent back to the client. The client’s TCP stack keeps retransmitting the SYN, gets no answer, and eventually gives up. The signature you see is a connection timeout — a long hang, then failure.
REJECT — The packet is discarded but the firewall sends back an explicit error (typically an ICMP “port unreachable” or a TCP RST). The client learns immediately that the door is closed. The signature is connection refused — a fast, clean failure.

DROP   → silence → client waits → TIMEOUT   (slow)
REJECT → error sent back        → REFUSED    (fast)

This difference is a genuine diagnostic lever, and it correlates with layers:

A connection timeout points at a DROP — the default behavior of most perimeter filters. Cloud security groups, network ACLs, and hardened host firewalls all silently drop by policy. A timeout says “something upstream is swallowing my packets.”
A connection refused points at a REJECT or, just as often, at nothing listening on the port — the packet reached a host but the port had no service, so the kernel sent a RST. Refused means “I reached the machine; the specific door is closed.”

So the shape of the failure narrows the search before you read a single rule. A slow timeout says “look outward — perimeter, security group, drop policy.” A fast refusal says “I got to the host — is the service even up, or is a local REJECT in play?”

🛠️ DevOps Perspective — “Connection timeout” across a security-group boundary and “connection refused” from a host firewall are different findings that point at different teams. A timeout to a service behind a cloud load balancer is very likely a security-group or network-ACL gap — a networking/platform ticket. A refusal on a port you own is very likely the app being down or a local REJECT — an app-team or host issue. Reporting the exact failure phrase, not just “it’s broken,” routes the problem to the right people the first time.

Locating the Blocking Layer with Nmap

You do not have to guess which layer drops your traffic — you can probe it. As covered in Nmap network validation, Nmap reports a port’s state, and the state distinguishes the two failure modes cleanly:

nmap -Pn -p 443 api.internal.example

open — Something is listening and reachable. The path is clear all the way through.
closed — The packets reached the host and got a RST back. The host is reachable; nothing is listening on that port (or a REJECT is in play). This maps to connection refused.
filtered — Nmap got no response at all — the probes were silently dropped. Something between you and the host (or its DROP-policy firewall) is swallowing the packets. This maps to connection timeout.

The distinction between closed (reachable host, no service / REJECT) and filtered (packets vanishing / DROP) is exactly the DROP-vs-REJECT fingerprint, made explicit by a tool. Run the same probe from different vantage points — outside the cloud, from another host in the subnet, from inside the container network — and watch where filtered turns into open. The layer at which the state changes is the layer doing the filtering. That is how you locate the blocker without touching a single rule.

Try It Yourself

🧪 Try It — Practice on infrastructure you own — a lab VM or the running kali-network-lab:

Run nft list ruleset (or iptables -L -n -v) and identify the policy on the INPUT chain — accept or drop?

From another host, run nmap -Pn -p 22,443,5432 <lab-ip> and note which ports come back open, closed, or filtered.

Correlate: does a filtered port match a drop policy with no matching rule? Does a closed port match a service that is not running?

Re-run iptables -L -n -v and watch the pkts counter on the matching rule climb — that is the packet you just sent, being counted.

You have now tied a client-side symptom (filtered/closed) to a server-side rule — the core skill of firewall diagnosis. Remember the ethical boundary: only scan, inspect, or test systems you own or have explicit permission to assess.

Common Problems

Wide-open host firewall, still timing out. The block is upstream — a cloud/OpenStack security group or a network ACL. Nothing on the host can show it; check the cloud console. The timeout (not refusal) is your clue it is a DROP at the perimeter.
Connection refused on a port you expect to work. Usually the service is not listening, not a firewall at all. Confirm with ss -ltnp (see ports and listening services) before blaming rules.
Containers have no internet. Look at the FORWARD chain and NAT, not INPUT. A drop FORWARD policy or a missing masquerade rule is the usual cause.
Pods suddenly cannot reach each other in Kubernetes. A new NetworkPolicy was applied — selecting a pod flips it to default-deny. Check kubectl get networkpolicy. The future “Kali Linux + Kubernetes” path will go deep on this; for now see the Kubernetes & Helm category.
Long-lived connections die randomly. Suspect a full or aggressively-timed conntrack table, not a rule you can read.
The urge to just flush the rules. Flushing “fixes” it by removing evidence and often opens the host wide — a security incident, not a diagnosis. Locate the layer instead.

Troubleshooting Workflow

Work the failure signature first, then the layers, from the outside in:

Application -> TLS -> Port -> DNS -> Gateway -> Route -> Interface
                       ^
             firewall filtering acts HERE:
             is the packet dropped (timeout)
             or rejected/closed (refused)?

Observe — Capture the exact symptom. Timeout or refused? Slow hang or instant failure? This alone splits DROP from REJECT.
Hypothesis — Timeout ⇒ a silent DROP at a perimeter layer (security group, ACL, drop policy). Refused ⇒ REJECT or nothing listening, on a reachable host.
Test — Probe with nmap -Pn from several vantage points; note open / closed / filtered at each. Watch where the state changes.
Evidence — On the host, nft list ruleset / iptables -L -n -v; read policies and check whether counters for the relevant rule increment when you probe.
Identify the layer — The point where filtered becomes reachable, or where the counter first stops incrementing, is the blocking layer: cloud SG, network ACL, host firewall, NetworkPolicy, or the app.
Correct — Fix at the right layer and the narrowest scope — only on systems you administer. Never blanket-flush.
Validate — Re-probe with Nmap, confirm the port is now open, watch the accept-rule counter climb, and confirm the application connection succeeds end to end.

For the reachability basics that precede this, revisit testing connectivity and Linux routing — a firewall is only the culprit once route and neighbor are ruled out.

What You Learned

A firewall is a packet filter that matches packets against ordered rules and accepts, drops, or rejects them at chain hook points.
INPUT filters traffic to this host, OUTPUT traffic from it, and FORWARD traffic passing through it — and on container/K8s hosts, FORWARD is usually where the action is.
Stateful firewalls use conntrack to allow ESTABLISHED/RELATED return traffic automatically, so you only write explicit rules for NEW connections.
nft list ruleset shows a modern nftables config (read each chain top-down, first match wins, note the policy); iptables -L -n -v shows the legacy view, where -n skips slow DNS lookups and -v reveals pkts/bytes counters that prove whether a rule is hit.
On modern systems iptables is usually a compatibility shim over nftables — both views may describe the same underlying rules, so check both.
Filtering is layered — cloud/OpenStack security groups, network ACLs, host firewalls, and Kubernetes NetworkPolicies — and a packet can die at any layer, including ones invisible from inside the host.
DROP yields a silent timeout (typical of perimeter layers); REJECT or a dead port yields an immediate refused — and Nmap’s filtered (DROP) vs closed (REJECT/no service) states, probed from multiple vantage points, locate the blocking layer without editing a rule.
Diagnose, do not flush: identify the layer and correct it narrowly, only on systems you own or administer.

You now have the full toolkit — interfaces, addressing, routing, ARP, ports, DNS, packet capture, Nmap, and firewalls. In Part 15, the capstone lab, we assemble the kali-network-lab with deliberate, safe misconfigurations across several of these layers and diagnose each one end to end — turning everything from this series into a single, repeatable troubleshooting practice. See you in the DevOps network troubleshooting lab.

Linux Network Troubleshooting: Firewalls and Filtering