🚨 Filebeat Error Guide: 'Error extracting container id' — Fix Container Log Path Parsing

Overview

Filebeat logs this when the container input (or the add_kubernetes_metadata processor) cannot pull a container id out of the log file’s path:

Error extracting container id - source value does not contain matcher's logs_path '/var/log/containers/'

On Kubernetes and Docker hosts, Filebeat derives the container id from the log file path so it can enrich each line with pod, namespace, and container metadata. The container input and the metadata matcher expect a specific path layout (e.g. /var/log/containers/<pod>_<ns>_<container>-<id>.log symlinked to the runtime’s log dir). If Filebeat is pointed at the wrong path, or the matcher’s logs_path doesn’t match the real layout, the id extraction fails and events arrive without Kubernetes metadata.

Symptoms

• Repeated Error extracting container id warnings in the Filebeat log.
• Container/pod events reach Elasticsearch but lack kubernetes.* fields (namespace, pod, labels).
• The problem appears after changing the container runtime (Docker -> containerd) or the log path.
• add_kubernetes_metadata logs does not contain matcher's logs_path.
• Some nodes enrich correctly and others don’t, due to differing log layouts.

Common Root Causes

• Input path doesn’t match the matcher’s logs_path — reading from /var/lib/docker/containers/ while the matcher expects /var/log/containers/.
• Wrong runtime log layout — containerd/CRI-O store logs differently from Docker’s json-file driver.
• Missing symlink directory — /var/log/containers/ (the symlink farm) not mounted into the Filebeat container.
• Custom stream/format on the container input not matching the runtime.
• Matcher misconfiguration — logs_path in add_kubernetes_metadata not aligned with the input paths.

Diagnostic Workflow

Confirm the real on-disk log path and layout, then align both the input and the metadata matcher to it.

ls -l /var/log/containers/ | head          # symlinks: <pod>_<ns>_<container>-<id>.log
ls -l /var/log/pods/ | head                # containerd/CRI layout
readlink -f /var/log/containers/*.log | head    # where the symlinks point

Use the container input pointed at the symlink farm, with the metadata matcher’s logs_path matching it:

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: container
    id: k8s-container-logs
    paths:
      - /var/log/containers/*.log     # the symlink farm the matcher expects
    stream: all                        # stdout + stderr

processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
        - logs_path:
            logs_path: "/var/log/containers/"   # MUST match the input path root

filebeat test config -c /etc/filebeat/filebeat.yml
filebeat -e -c /etc/filebeat/filebeat.yml -d "kubernetes,input"   # watch id extraction + enrichment

Example Root Cause Analysis

A cluster migrated from Docker to containerd. The old Filebeat config still read directly from /var/lib/docker/containers/*/*.log, but add_kubernetes_metadata used the default matcher logs_path: /var/log/containers/. Because the input path and the matcher path no longer agreed, every line produced Error extracting container id - source value does not contain matcher's logs_path '/var/log/containers/', and pods showed up with no Kubernetes metadata.

The fix was to point the container input at the standard symlink farm that both runtimes populate, so the id could be parsed from the filename:

- type: container
  id: k8s-container-logs
  paths: ["/var/log/containers/*.log"]

They also mounted /var/log/containers and /var/log/pods (read-only) into the Filebeat DaemonSet so the symlinks and their targets were both visible. After redeploy, ids extracted cleanly and kubernetes.* fields returned.

Prevention Best Practices

• Always read container logs from /var/log/containers/*.log, the runtime-agnostic symlink farm, not the raw Docker/containerd directories.
• Keep the input paths root and the add_kubernetes_metadata logs_path matcher in sync.
• In DaemonSets, mount both /var/log/containers (symlinks) and /var/log/pods (targets) read-only.
• Re-verify the log layout after any container-runtime change (Docker -> containerd -> CRI-O).
• Use the container input type rather than a raw filestream for container logs so parsing/enrichment is built in.

Quick Command Reference

ls -l /var/log/containers/ | head          # inspect symlink farm
readlink -f /var/log/containers/*.log       # find real log targets
filebeat test config                        # validate config
filebeat -e -d "kubernetes,input"           # watch id extraction + enrichment
# align: container input paths root  ==  add_kubernetes_metadata logs_path

Conclusion

Error extracting container id means Filebeat can’t parse a container id from the log path, so Kubernetes metadata enrichment fails. The root cause is nearly always a mismatch between where Filebeat reads logs and where the add_kubernetes_metadata matcher expects them. Point the container input at the standard /var/log/containers/*.log symlink farm, keep the matcher’s logs_path identical, and mount both the symlinks and their targets into the DaemonSet. Re-check the layout after any runtime change and enrichment stays intact.