🚨 Filebeat Error: 'Connection marked as failed because the onConnect callback failed' — Cause, Fix, and Troubleshooting Guide

Overview

When Filebeat’s Elasticsearch output establishes a connection, it runs an onConnect callback that performs setup work — checking the ES version, loading the index template, and installing ingest pipelines. If any of that setup fails, the whole connection is marked failed and Filebeat backs off:

Connection marked as failed because the onConnect callback failed: Error loading pipeline for fileset nginx/access: This module requires an Elasticsearch plugin that provides the geoip processor. Please visit the Elasticsearch documentation for instructions on how to install this plugin.

The TCP/TLS connection and authentication succeeded — the failure is in the post-connect callback. The nested reason is the real error: a template load rejection, a pipeline that references a missing processor, or a version-compatibility check. Until the callback succeeds, no events publish over that connection.

Symptoms

• Connection marked as failed because the onConnect callback failed: <reason> on startup or reconnect.
• Authentication passes (filebeat test output connects) but publishing never begins.
• The nested reason mentions a pipeline, a missing processor (geoip, user_agent), or a template.
• The error repeats with backoff; no documents index in the meantime.

Common Root Causes

• Pipeline references a missing processor — e.g. geoip/user_agent on a stripped-down ES.
• Template load rejected — insufficient privileges to put the index template.
• Version mismatch — Filebeat major version newer/older than the ES cluster it connects to.
• Ingest node disabled on the target node, so pipeline install fails.
• Transient ES error during the setup call (5xx) that fails the callback.

How to diagnose

Read the nested reason in the foreground — it names the failing setup step:

filebeat -e -c /etc/filebeat/filebeat.yml | grep -i 'onConnect\|pipeline\|template'

Confirm the ES version is compatible and ingest is available:

curl -sk https://es01:9200 -u elastic:$ES_PASS | jq '.version.number'
curl -sk 'https://es01:9200/_nodes/ingest?filter_path=nodes.*.ingest.processors.*.type' \
  -u elastic:$ES_PASS | grep -o 'geoip\|user_agent'

Fixes

If a module pipeline needs a processor your cluster lacks, either install it or disable that module’s pipeline. The geoip and user_agent processors ship with Elasticsearch by default in 7.x+; a missing one usually means a stripped image:

# On the Elasticsearch node, confirm the ingest-geoip module is present
bin/elasticsearch-plugin list

Load templates and pipelines with a privileged user via setup instead of at connect time:

filebeat setup --pipelines --index-management -e \
  -E 'output.elasticsearch.username=elastic' \
  -E 'output.elasticsearch.password=${ES_PASS}'

Match Filebeat’s major version to the cluster and grant the output user template privileges:

# minimum privileges for the output user (via ES role)
# cluster: ["monitor","manage_ilm","manage_index_templates"]
# indices: [{"names":["filebeat-*"],"privileges":["create_doc","create_index","view_index_metadata"]}]

What to watch out for

• The connection succeeds before the callback — this is not an auth or network problem.
• Running filebeat setup with an admin user avoids needing template privileges on the shipping user.
• Keep Filebeat and Elasticsearch on the same major version to avoid compatibility check failures.
• A missing geoip/user_agent processor is almost always a custom/minimal ES image.

Related

• Filebeat Error: ‘Filebeat is unable to load the ingest pipelines’
• Filebeat Error Guide: ‘could not load index template’
• Filebeat Error: ‘pipeline with id does not exist’