🚨 Filebeat Error: 'pipeline with id does not exist' — Cause, Fix, and Troubleshooting Guide

Overview

When Filebeat routes events through an Elasticsearch ingest pipeline — via a module, output.elasticsearch.pipeline, or a pipeline on the input — Elasticsearch must have that pipeline installed. If the named pipeline is absent, it rejects the write:

Failed to perform any bulk index operations: 400 Bad Request: {"error":{"type":"illegal_argument_exception","reason":"pipeline with id [filebeat-8.13.0-nginx-access-pipeline] does not exist"}}

The document reached Elasticsearch but named a pipeline the cluster does not know. This happens when module pipelines were never loaded, when filebeat setup --pipelines ran against a different cluster, or when a manually configured pipeline: name is misspelled or was deleted. The events are non-indexable until the pipeline exists.

Symptoms

• pipeline with id [...] does not exist in a 400 Bad Request bulk error.
• Only datasets that use a pipeline fail; plain inputs still index.
• Appeared after enabling a module without re-running setup, or after a cluster rebuild.
• GET _ingest/pipeline/<id> returns 404 for the named pipeline.

Common Root Causes

• Pipelines never loaded — filebeat setup --pipelines was skipped.
• Wrong cluster — setup ran against a different ES than the output targets.
• Module enabled after setup — its pipeline was never installed.
• Misspelled/renamed pipeline: in the output or input config.
• Pipeline deleted during a cluster reset or cleanup job.

How to diagnose

Check whether the named pipeline exists on the target cluster:

curl -sk 'https://es01:9200/_ingest/pipeline/filebeat-8.13.0-nginx-access-pipeline?pretty' \
  -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_ingest/pipeline?pretty' -u elastic:$ES_PASS | grep filebeat

See what pipeline Filebeat references and which modules are enabled:

grep -i 'pipeline' /etc/filebeat/filebeat.yml
filebeat modules list

Fixes

Load the module/ingest pipelines against the cluster the output writes to:

filebeat setup --pipelines -e \
  -E 'output.elasticsearch.hosts=["https://es01:9200"]' \
  -E 'output.elasticsearch.username=elastic' \
  -E 'output.elasticsearch.password=${ES_PASS}'

For a specific module, load only its pipelines:

filebeat setup --pipelines --modules nginx
systemctl restart filebeat

If you set a custom pipeline, make sure the name matches an installed pipeline exactly:

output.elasticsearch:
  hosts: ["https://es01:9200"]
  pipeline: "my-custom-pipeline"   # must exist: PUT _ingest/pipeline/my-custom-pipeline

What to watch out for

• Pipelines are cluster-scoped — re-run --pipelines after any cluster rebuild.
• Enabling a module later does not auto-load its pipeline; re-run setup.
• The pipeline name is version-stamped (e.g. filebeat-8.13.0-...); an upgrade changes it.
• A misspelled custom pipeline: produces the same 404 as a missing one.

Related

• Filebeat Error: ‘Filebeat is unable to load the ingest pipelines’
• Filebeat Error: ‘Connection marked as failed because the onConnect callback failed’
• Filebeat Error Guide: ‘mapper_parsing_exception’