🔬 eBPF Security Observability: Seeing What Your Kernel Actually Does

For most of my career, getting deep visibility into what a Linux box was really doing meant a painful trade. You could run heavyweight agents that hooked everything and tanked your performance, or you could run lightweight log shippers that missed the interesting stuff happening in kernel space. eBPF collapsed that trade-off, and it’s the most important thing to happen to security observability in a decade.

The short version: eBPF lets you run small, sandboxed, verified programs inside the Linux kernel, attached to events — syscalls, network packets, function calls — without modifying kernel source or loading risky modules. For security, that means you can watch process execution, network connections, and file access from the one vantage point nothing can hide from: the kernel itself.

Why the kernel is the right place to watch

Userspace monitoring has a fundamental weakness: an attacker who gains enough privilege can tamper with it. They can kill your agent, swap a binary, or hide a process. The kernel sees the ground truth — every execve, every connect, every open — and a properly attached eBPF program observes those events before userspace tricks can obscure them.

And it’s cheap. eBPF programs run in a restricted virtual machine that the kernel verifies before loading: it proves the program terminates, doesn’t access invalid memory, and can’t crash the kernel. You get kernel-level insight with overhead measured in single-digit percentages, which is why you can leave it running in production permanently.

Three things worth watching

You don’t need to write raw eBPF to benefit — tools like the bcc suite and bpftrace give you a high-level entry point. A few security-relevant one-liners show the shape of it.

Every process that executes, with parentage:

# Trace all exec() calls cluster-wide; -P shows parent
execsnoop-bpfcc -P

This is the single most useful security trace there is. Attacks almost always involve spawning a process — a shell, a downloader, a recon tool — and execsnoop shows you each one with its full command line and parent process. Suspicious chains (a web server spawning curl spawning bash) jump right out.

Every outbound TCP connection:

# Trace TCP connects with PID, command, destination
tcpconnect-bpfcc

Exfiltration and command-and-control both require the network. Watching connect() at the kernel level catches the unexpected outbound flow regardless of how the process tried to hide it.

Files being opened, filtered to what matters:

# Watch for reads of sensitive credential files
opensnoop-bpfcc -e | grep -E 'shadow|id_rsa|\.aws/credentials|token'

A custom probe in bpftrace is just as readable when you want something specific — here, anyone touching the kubelet’s service-account tokens:

// Alert on opens of service-account token files
tracepoint:syscalls:sys_enter_openat
/ str(args->filename) == "/var/run/secrets/kubernetes.io/serviceaccount/token" /
{
    printf("token read by pid %d (%s) uid %d\n", pid, comm, uid);
}

From ad-hoc traces to a real platform

bpftrace one-liners are perfect for investigation, but for continuous security observability you want a platform that does the kernel attachment, enrichment, and policy for you. Tetragon (from the Cilium project) is the leading security-focused one. It lets you declare what to observe and even enforce, as a Kubernetes resource:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: monitor-sensitive-files
spec:
  kprobes:
  - call: "security_file_open"
    syscall: false
    args:
    - index: 0
      type: "file"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/shadow"
        - "/root/.ssh"

Tetragon enriches every event with pod, container, and process-ancestry context, so an alert is immediately actionable. Critically, eBPF can also enforce, not just observe — a policy can kill a process the instant it performs a forbidden action, in-kernel, before the syscall completes. That’s a powerful capability, and exactly the kind you introduce carefully: start in observe mode, prove the policy doesn’t catch legitimate behavior, and only then let it block.

The network side: visibility you couldn’t get before

eBPF also transformed network observability. Because it sees packets at the kernel, it reconstructs service-to-service flows with identity — which pod talked to which pod, on what port, allowed or denied — without sidecars or packet capture. Cilium’s Hubble surfaces exactly this:

# Live flow visibility, filtered to drops
hubble observe --verdict DROPPED

A wall of DROPPED flows to your database from a workload that shouldn’t be talking to it is a breach indicator you’d never see in application logs. This is the data that makes network segmentation auditable instead of aspirational.

Operating it responsibly

A few things I’ve learned running eBPF security tooling in anger:

• Mind your kernel versions. eBPF capabilities depend heavily on kernel version. Standardize on a recent kernel across the fleet or you’ll have features that work on some nodes and not others.
• Cardinality will bite you. execsnoop on a busy node is a lot of events. Filter at the source (in the eBPF program) rather than shipping everything and filtering later, or you’ll drown your pipeline.
• Observe before enforce, always. The enforcement capability is genuinely exciting and genuinely dangerous. A kill-on-match policy that’s wrong is an outage. Earn confidence in observe mode first.

eBPF gives you the deepest, cheapest security visibility available on Linux today — the kernel’s own view, programmable and safe. Start with execsnoop and tcpconnect on one node to feel the signal, graduate to a platform like Tetragon or Hubble for continuous coverage, and reach for enforcement only once observation has earned your trust. It complements runtime detection and segmentation nicely — the broader security hardening guides cover how these layers reinforce each other, and putting new tracing policies through automated code review keeps an over-broad enforcement selector from shipping unnoticed.

eBPF probes and tracing policies are starting points. Validate kernel compatibility and run enforcement policies in observe mode against real workloads before letting them block anything.