Log Shipping & Observability Pipeline Errors
Log and metrics pipelines fail quietly: Filebeat stops harvesting or replays a file, Logstash wedges on a bad grok pattern or a blocked output, and Loki rejects writes or cannot reach its object store. The symptom is usually 'logs stopped arriving,' and the cause is somewhere along the pipeline rather than in one component. Each guide opens with the direct cause and the config and diagnostic commands to confirm it. Start with your symptom below, or paste the agent log into the assistant.
Fix your error now
Paste the error or logs and let the Incident Assistant diagnose it, or run a guided triage prompt with AI — no copy-paste.
Every guide in this cluster
150 guides-
'too many open files' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'too many open files': raise the file-descriptor limit and cap harvesters so Filebeat can open its logs, registry, and output sockets.
-
'Logstash could not be started because there is already another instance' — Clear the data.lock
Fix Logstash 'could not be started because there is already another instance': find the running PID, clear a stale .lock, or set a unique path.data.
-
'failed parsing config' — Fix YAML, Renamed Fields, and Env Expansion
Fix Loki 'failed parsing config: field not found in type': catch YAML typos, renamed fields, and env vars, then run -verify-config.
-
'registry file is corrupt' — Recover a Broken Filebeat Registry
Fix Filebeat 'registry file is corrupt': diagnose truncated or malformed registry state, reset the registry safely, and prevent duplicate log ingestion.
-
'401 Unauthorized' — Restore Elasticsearch Output Auth
Fix Filebeat '401 Unauthorized unable to authenticate user': repair output credentials, API keys, and role privileges so the bulk request authenticates and events index into Elasticsearch again.
-
'413 Request Entity Too Large' — Shrink the Bulk Request
Fix Filebeat bulk '413 Request Entity Too Large': reduce bulk size, raise Elasticsearch http.max_content_length or the proxy body limit, and trim oversized events so bulk requests are accepted.
-
'429 Too Many Requests' — Relieve Bulk Backpressure
Fix Filebeat bulk '429 Too Many Requests' from Elasticsearch: understand write thread-pool rejections, tune bulk size and workers, and scale the cluster so backpressure stops throttling ingest.
-
'503 Service Unavailable' — Recover a Blocked Output
Fix Filebeat bulk '503 Service Unavailable' from Elasticsearch: diagnose a red or recovering cluster, blocked indices, and proxy outages, then restore writes and drain the queue safely.
-
'add_docker_metadata: unable to connect to Docker daemon' — Fix Container Enrichment
Fix Filebeat 'add_docker_metadata: unable to connect to Docker daemon': mount the socket, fix permissions, or switch to Kubernetes metadata.
-
'backoff(elasticsearch(...)): lookup: no such host' — Fix Output DNS Resolution
Fix Filebeat 'backoff(elasticsearch(...)): lookup es01: no such host': repair DNS, hostnames, and endpoints so the output resolves and events ship.
-
'Can only start an input when all related states are finished' — Fix Overlapping Inputs
Fix Filebeat 'Can only start an input when all related states are finished': resolve overlapping input paths, duplicate harvesters, and stuck file states.
-
'connection refused' — Restore the Elasticsearch Output
Fix Filebeat 'Failed to connect to backoff(elasticsearch(...)): connection refused': verify the output host and port, TLS scheme, firewall rules, and that Elasticsearch is actually listening.
-
'connection reset by peer' to Logstash — Stabilize the Beats Output
Fix Filebeat 'Failed to connect to Logstash: connection reset by peer': align the Beats input, TLS, and congestion settings so the Lumberjack channel to Logstash stops dropping mid-stream.
-
'could not load index template' — Fix Template Setup
Fix Filebeat 'could not load index template': resolve permission, connectivity, and template-conflict failures during setup so mappings load and your data stream indexes correctly.
-
'could not start registrar: Error loading state' — Fix Registry Load Failures
Fix Filebeat 'could not start registrar: Error loading state': repair or reset a broken registry, fix data-path permissions, and recover from unclean shutdowns.
-
'data path already locked by another beat' — Fix a Contended path.data
Fix Filebeat 'data path already locked by another beat': stop duplicate instances, give each Beat its own path.data, and clear stale lock files.
-
'dial tcp: i/o timeout' — Fix a Silently Dropped Output
Fix Filebeat 'dial tcp ...: i/o timeout' to Elasticsearch: distinguish dropped packets from refused connections, check firewalls, security groups, MTU, and DNS so the output can reach the cluster.
-
'Error creating runner from config' — Fix Invalid Input Settings
Fix Filebeat 'Error creating runner from config': resolve invalid input options, wrong types, missing fields, and processor errors that block input startup.
-
'Error decoding JSON' — Fix JSON Log Parsing Failures
Fix Filebeat 'Error decoding JSON': handle malformed lines, multiline JSON, wrong parser settings, and key collisions so structured logs parse cleanly.
-
'Error extracting container id' — Fix Container Log Path Parsing
Fix Filebeat 'Error extracting container id': correct container input paths and add_kubernetes_metadata so container logs parse and enrich correctly.
-
'error loading config file ... yaml: line X' — Fix YAML Syntax Errors
Fix Filebeat 'error loading config file ... yaml: line X': resolve indentation, tabs, unquoted specials, and mapping errors that stop Filebeat from starting.
-
'error loading index pattern' — Fix Kibana Setup
Fix Filebeat 'Exiting: error loading index pattern': resolve Kibana connectivity, auth, and version issues during filebeat setup so dashboards and the index pattern load cleanly.
-
'Failed to create input: unknown input type' — Fix Invalid Input Types
Fix Filebeat 'failed to create input: unknown input type': correct misspelled or version-removed input types, casing slips, and wrong module input names.
-
'harvester ... file was truncated' — Fix Truncated Log Handling
Fix Filebeat 'file was truncated' warnings: handle in-place truncation, copytruncate rotation, and offset resets so Filebeat re-reads files correctly.
-
'Harvester could not be started ... permission denied' — Fix Log File Access
Fix Filebeat 'Harvester could not be started ... permission denied': grant read access to log files and directories, fix ownership, ACLs, and SELinux.
-
'id already exists in the filestream input' — Fix Duplicate Input IDs
Fix Filebeat 'id already exists in the filestream input': assign unique filestream ids, resolve copy-paste duplicates, and avoid state collisions.
-
'ILM policy not found' — Restore Lifecycle Management
Fix Filebeat 'ILM policy not found': load the lifecycle policy, align template and rollover alias, and get index lifecycle management working so Filebeat indices roll over as expected.
-
'mapper_parsing_exception' — Fix Field Parse Failures
Fix Filebeat 'mapper_parsing_exception failed to parse field': resolve type conflicts where a string lands in a numeric or date field, using dropped-event logs, mappings, and ingest fixes.
-
'mapping conflict' — Resolve Field Type Clashes
Fix Filebeat 'illegal_argument_exception mapper cannot be changed' mapping conflicts: reconcile a field mapped two ways across events, using templates, ingest renames, and reindexing.
-
'multiline pattern did not match' — Fix Stack Trace Grouping
Fix Filebeat multiline pattern issues: correct the regex, negate, and match settings so stack traces group into single events instead of splitting.
-
'temporary bulk send failure' — Stabilize the Bulk Pipeline
Fix Filebeat 'Failed to publish events: temporary bulk send failure': diagnose partial bulk rejections, dropped connections, and overloaded Elasticsearch, then tune bulk size and retries.
-
'x509: certificate signed by unknown authority' — Trust the CA
Fix Filebeat 'x509: certificate signed by unknown authority' to Elasticsearch: supply the correct CA, fix cert chains and hostnames, and stop the TLS handshake from failing on the output.
-
'circuit_breaking_exception: Data too large' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'circuit_breaking_exception Data too large': shrink bulk requests and relieve Elasticsearch heap pressure so the parent breaker stops tripping.
-
'cluster_block_exception ... index read-only / allow delete (api)' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'cluster_block_exception FORBIDDEN/12/index read-only / allow delete': free disk past the flood-stage watermark and clear the write block.
-
'config file ... can only be writable by the owner' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'config file ... can only be writable by the owner but the permissions are': set 0600/0644 ownership so Filebeat loads its config.
-
'Connection marked as failed because the onConnect callback failed' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Connection marked as failed because the onConnect callback failed': repair the template, pipeline, or version check that runs on connect.
-
'dataset ... has no template' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'dataset ... has no template': load the correct index template or data stream so events for the dataset index instead of being rejected.
-
'Failed to publish events caused by: EOF' to Logstash — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Failed to publish events caused by: EOF' to Logstash: stop the Beats input closing the connection via back-pressure, TLS, or timeout.
-
'Failed to publish events caused by: read tcp ... i/o timeout' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Failed to publish events caused by: read tcp ... i/o timeout': raise output timeout and shrink bulk size when ES is slow to respond.
-
'Failed to start crawler' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Failed to start crawler': resolve the bad input, glob, or module that stops the crawler and prevents any harvesters from starting.
-
'Filebeat is unable to load the ingest pipelines' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'unable to load the ingest pipelines': grant ingest privileges, target an ingest node, and re-run setup so module pipelines install.
-
'Harvester crawler could not be started' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Harvester could not be started': resolve open-file limits, symlink loops, and read errors that stop a harvester from reading a log file.
-
'illegal_argument_exception' on bulk index — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'illegal_argument_exception' on bulk index: resolve field limits, bad values, and pipeline issues that make Elasticsearch reject events.
-
'Index management requested but the Elasticsearch output is not configured/enabled' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'Index management requested but the Elasticsearch output is not configured/enabled': run setup against ES when shipping through Logstash.
-
'index_closed_exception' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'index_closed_exception closed': reopen the target index or fix ILM/rollover so bulk writes stop hitting a closed Elasticsearch index.
-
'kafka: client has run out of available brokers to talk to' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'kafka: client has run out of available brokers to talk to': repair broker addresses, advertised listeners, TLS, and SASL for the Kafka output.
-
'No paths were defined for input' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'No paths were defined for input accessing config': add a valid paths list (or fix indentation) so the input initializes and harvests logs.
-
'Non-zero metrics in the last 30s' — Cause, Fix, and Troubleshooting Guide
Understand Filebeat 'Non-zero metrics in the last 30s': read the periodic metrics line to spot stalled harvesters, output failures, and dropped events.
-
'pipeline with id does not exist' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'pipeline with id [...] does not exist': install the referenced ingest pipeline or fix the output pipeline setting so bulk writes succeed.
-
'State for ... not removed, file still open' — Cause, Fix, and Troubleshooting Guide
Understand Filebeat 'State for file not removed, file still open': tune close and clean options so registry state clears after rotation instead of piling up.
-
'version_conflict_engine_exception' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'version_conflict_engine_exception': stop duplicate document IDs and double-shipping so bulk writes stop colliding on the same _id.
-
'x509: certificate is valid for ..., not ...' — Cause, Fix, and Troubleshooting Guide
Fix Filebeat 'x509: certificate is valid for X, not Y': align the output host with the certificate SAN so TLS verification of Elasticsearch succeeds.
-
'_dateparsefailure' Tag on Events — Match Every Timestamp Format
Fix the '_dateparsefailure' tag in Logstash: understand the tag's meaning, align timezone and locale, and match multiple timestamp formats in one date filter.
-
'_grokparsefailure' — Make Your Grok Filter Match
Fix the _grokparsefailure tag in Logstash: see why grok did not match, test patterns, handle variant log formats, and stop silent parse failures.
-
'circuit_breaking_exception ... Data too large' — Relieve the Elasticsearch Memory Breaker
Fix Logstash 'circuit_breaking_exception: Data too large' from the Elasticsearch output: shrink bulk requests, cut fielddata pressure, and add ES heap.
-
'Could not execute action: PipelineAction::Create' — Fix the Failing Pipeline Config
Fix Logstash 'Could not execute action: PipelineAction::Create': read the wrapped error, fix plugin syntax, missing plugins, and bad settings, then reload.
-
'Could not index event to Elasticsearch ... mapper_parsing_exception' — Fix the Type Conflict
Fix Logstash 'Could not index event to Elasticsearch ... mapper_parsing_exception': coerce field types, control dynamic mapping, and capture rejects in the DLQ.
-
'could not load driver class com.mysql.jdbc.Driver' — Fix the JDBC Input Driver
Fix Logstash jdbc input 'could not load driver class': set jdbc_driver_library and jdbc_driver_class correctly, verify the JAR path, and test it.
-
'dictionary file not found' — Fix a Missing translate dictionary_path
Fix Logstash translate filter 'dictionary file not found': verify dictionary_path, file permissions, YAML/CSV format, and the refresh settings.
-
'Elasticsearch Unreachable' — Restore the Output Connection
Fix Logstash 'Attempted to resurrect connection ... Elasticsearch Unreachable': check hosts, DNS, ports, TLS and auth, then confirm the output recovers.
-
'Failed parsing date from field' — Fix the date Filter Pattern
Fix Logstash date filter 'Failed parsing date from field' and _dateparsefailure: match the exact timestamp format, handle locale and timezone, test it.
-
'Invalid FieldReference' — Fix Malformed [foo][bar] Field Syntax
Fix Logstash 'Invalid FieldReference' errors: correct unbalanced brackets, bad [foo][bar] syntax, and sprintf names that break the field parser.
-
'java.io.IOException: Page file ... different size' — Recover a Corrupt Persistent Queue
Fix Logstash startup 'java.io.IOException: Page file ... different size': diagnose persistent-queue corruption, reset the queue, and prevent recurrence.
-
Java heap space' — Right-Size the JVM Heap
Fix Logstash 'java.lang.OutOfMemoryError: Java heap space': right-size Xms/Xmx, cut batch size and workers, move lookups off-heap, and capture heap dumps.
-
'JSON parse error, original data now in message field' — Fix the json Filter
Fix Logstash 'JSON parse error, original data now in message field' and _jsonparsefailure: handle malformed JSON, wrong source fields, split events.
-
'Limit of total fields [1000] has been exceeded' — Stop the Mapping Explosion
Fix Logstash 'Limit of total fields [1000] has been exceeded': find the high-cardinality keys, control dynamic mapping, and prune fields before they explode.
-
'No configuration found in the configured sources' — Fix path.config and pipelines.yml
Fix Logstash 'No configuration found in the configured sources': point path.config at real .conf files, reconcile pipelines.yml vs -f, and fix globs.
-
'pattern %{FOO} not defined' — Fix Grok Compile Failures
Resolve 'pattern not defined' and 'Grok compile failed' in Logstash: register custom patterns, fix typos and paths, and stop start-up crashes.
-
'Pipeline aborted due to error' — Fix the Fatal Filter or Config Fault
Fix Logstash 'Pipeline aborted due to error': trace the Ruby/plugin exception, correct the bad config or filter, and stop the pipeline from crash-looping.
-
'Pipeline is blocked' — Diagnose and Clear Output Backpressure
Fix Logstash pipeline backpressure and stalled 'in-flight events': find the slow output, tune workers and batch size, and use a persistent queue to spill.
-
'PKIX path building failed' — Fix the Elasticsearch Output TLS Trust
Fix Logstash 'SSLHandshakeException: PKIX path building failed' on the Elasticsearch output: add the ES CA to the truststore and fix cacert settings.
-
'Received an event that has a different character encoding' — Fix the Codec Charset
Fix 'Received an event that has a different character encoding' in Logstash: align input codec charset, handle invalid UTF-8 bytes, and stop mojibake at ingest.
-
'retrying failed action with response code: 429' — Handle Elasticsearch Backpressure
Fix Logstash Elasticsearch output 'retrying failed action ... response code: 429': relieve write-queue saturation and tune batch size, workers, and shards.
-
'Ruby exception occurred' — Fix NoMethodError for nil in the ruby Filter
Fix Logstash 'Ruby exception occurred: undefined method for nil' in the ruby filter: guard nil fields, use event.get/set correctly, and test inline code safely.
-
'the dead_letter_queue is full' — Drain and Size the DLQ
Fix Logstash 'the dead_letter_queue is full': raise dead_letter_queue.max_bytes, drain the DLQ with a reader pipeline, and fix the failing ES writes.
-
'the pipeline is blocked, temporary refusing new connection' — Relieve Beats Input Backpressure
Fix Logstash beats input 'the pipeline is blocked, temporary refusing new connection': diagnose backpressure, tune workers, stop Filebeat refusals.
-
'version conflict, document already exists (409)' — Fix Duplicate document_id Writes
Fix Logstash Elasticsearch output 'version conflict, document already exists (409)': fix document_id, choose create vs index, and use idempotent writes.
-
Beats Input 'Connection reset by peer' — Stabilize Filebeat
Fix 'connection reset by peer' between Filebeat and the Logstash beats input: tune timeouts, resolve back-pressure and TLS mismatches, stop drops.
-
'_jsonparsefailure' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '_jsonparsefailure' from the json filter: guard non-JSON input, fix multiline framing, and use skip_on_invalid_json.
-
'Address already in use' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.inputs.tcp] BindException: Address already in use': find the process holding the port and give each input a unique port.
-
'Connection refused' — Cause, Fix, and Troubleshooting Guide
Fix Logstash jdbc input 'ConnectException: Connection refused': verify the database host, port, listener, and firewall in jdbc_connection_string.
-
'Could not create the Java Virtual Machine' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'Could not create the Java Virtual Machine' / Unrecognized VM option: clean up jvm.options and use the bundled JDK.
-
'Couldn't find any filter plugin named' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'Couldn't find any filter plugin named': install the missing plugin or correct the plugin name in your config.
-
'Failed to flush outgoing items' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.outputs.elasticsearch] Failed to flush outgoing items': address ES timeouts, batch size, and cluster load.
-
'Failed to install template' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.outputs.elasticsearch] Failed to install template': grant template privileges, check connectivity, or set manage_template.
-
'FORBIDDEN/12/index read-only / allow delete' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'FORBIDDEN/12/index read-only / allow delete (api)': free ES disk and clear the read-only block from the flood-stage watermark.
-
'Got response code '401'' — Cause, Fix, and Troubleshooting Guide
Fix Logstash elasticsearch output '401' unauthorized: supply valid user/password or api_key so the output can authenticate.
-
'Logstash stopped processing because of an error: (SystemExit)' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '(SystemExit) exit' FATAL at startup: read the preceding error, validate logstash.yml, and check path.data permissions.
-
'LogStash::ConfigurationError' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'LogStash::ConfigurationError': read the line/column, balance braces and quotes, and validate with logstash -t before restart.
-
'logstash.licensechecker' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.licensechecker] Unable to retrieve license': set valid X-Pack monitoring hosts, credentials, and license.
-
'Multiline codec with beats input is not supported' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'Multiline codec with beats input is not supported': remove the beats codec and do multiline in Filebeat instead.
-
'No space left on device' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'java.io.IOException: No space left on device': free disk and cap the persistent and dead letter queue sizes.
-
'org.apache.kafka.common.errors.TimeoutException' — Cause, Fix, and Troubleshooting Guide
Fix Logstash kafka 'org.apache.kafka.common.errors.TimeoutException': verify bootstrap_servers, brokers, topic metadata, and SASL/SSL settings.
-
'OutOfDirectMemoryError' — Cause, Fix, and Troubleshooting Guide
Fix Logstash beats 'io.netty OutOfDirectMemoryError': raise MaxDirectMemorySize, reduce Filebeat batch size, and load-balance connections.
-
'Permission denied' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.inputs.file] Permission denied': give the logstash user read access to logs and a writable sincedb_path.
-
'Pipeline worker error, the pipeline will be stopped' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.javapipeline] Pipeline worker error, the pipeline will be stopped': isolate the failing filter or output from the backtrace.
-
'retrying failed action with response code: 403' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'retrying failed action with response code: 403' security_exception: grant the writer role index privileges to authorize bulk writes.
-
'The MaxMind database has been expired' — Cause, Fix, and Troubleshooting Guide
Fix Logstash geoip 'MaxMind database has been expired': restore GeoIP auto-updates or pin a managed database on air-gapped hosts.
-
'Timeout executing grok' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '[logstash.filters.grok] Timeout executing grok': tame catastrophic regex backtracking with anchors, dissect, and timeout_millis.
-
'Too many open files' — Cause, Fix, and Troubleshooting Guide
Fix Logstash '(Errno::EMFILE) Too many open files': raise LimitNOFILE and use close_older/ignore_older on the file input.
-
'Unable to reload pipeline' — Cause, Fix, and Troubleshooting Guide
Fix Logstash 'Unable to reload pipeline': validate the new config with -t and handle non-reloadable plugins before auto-reload.
-
'502 Bad Gateway' — Recover the Gateway-to-Loki Upstream
Fix Loki '502 Bad Gateway': restore crashed upstream pods, correct the gateway proxy_pass, and raise proxy buffers and timeouts.
-
'AccessDenied: Access Denied status code: 403' — Fix Object Storage Permissions
Fix Loki S3/GCS 'AccessDenied status code: 403': diagnose IAM policy, bucket name, region, and credential problems that block chunk flushes and index reads, then grant the exact permissions Loki needs.
-
'at least 1 live replicas required, could only find 0' — Restore Ingester Write Quorum
Fix Loki 'at least 1 live replicas required, could only find 0': restore healthy ingesters, forget dead ring entries, and open gRPC 9095.
-
'bind: address already in use' — Free the Port or Change the Listener
Fix Loki 'listen tcp :3100: bind: address already in use': free the port, change http_listen_port or grpc_listen_port, and avoid collisions.
-
'chunk not found' — Reconcile the Index with What Actually Lives in Object Storage
Fix Loki's 'chunk not found': the index references a chunk that is missing from object storage. Align buckets, stop rogue retention, and clear stale caches.
-
'context canceled' — Trace Client Disconnects and Cancelled Sub-Queries
Fix Loki's 'context canceled': tell client disconnects and proxy resets from internal sub-query cancellation, then fix the real proxy or shard cause.
-
'context deadline exceeded' — Fix Query Timeouts on the Read Path
Resolve Loki's 'context deadline exceeded': raise query_timeout and http_server_write_timeout, narrow ranges, add label matchers, and speed slow object-storage reads causing timeouts.
-
'empty ring' / 'too many unhealthy instances in the ring' — Recover the Hash Ring
Fix Loki's 'empty ring' and 'too many unhealthy instances in the ring': repair KV store membership, clear stale ingesters, align replication_factor, and restore the read/write path.
-
'entry out of order' / 'entry too far behind' — Fix Timestamp Ordering at Ingest
Resolve Loki's 'entry out of order' and 'entry too far behind': fix clock skew, split streams by instance, tune max_chunk_age and reject_old_samples, and stop dropped log lines.
-
'error getting ring ... connection refused' — Fix the KV Store Backend
Fix Loki 'error getting ring ... connect: connection refused': repair the Consul/etcd KV backend, open the port, or switch to memberlist.
-
'failed to compact files' — Get the Compactor Running and Retention Working
Fix Loki's compactor 'failed to compact files' / 'failed to run compaction': run exactly one compactor, grant s3 delete for retention, and match the schema.
-
'failed to flush chunks' — Fix Object Storage Writes and Ingester Backpressure
Resolve Loki's 'failed to flush chunks': fix object-storage credentials and connectivity, relieve ingester backpressure, tune flush queues and chunk age, and stop WAL and memory growth.
-
'failed to replay WAL' — Recover an Ingester From a Corrupt Write-Ahead Log
Fix Loki's 'failed to replay WAL' and WAL corruption on startup: let the ingester truncate the bad tail, enable clean shutdowns, and lean on replication.
-
'flag provided but not defined' — Fix Renamed and Removed CLI Flags
Fix Loki 'flag provided but not defined': list valid flags with loki -help, move the setting into config, and align the image version.
-
'has 25 label names; limit 15' — Cut Label Cardinality Before the Distributor Rejects Your Push
Fix Loki's 'stream has 25 label names; limit 15': drop excess labels in your agent, move dynamic fields into the log line, and tune max_label_names_per_series.
-
'http: server gave HTTP response to HTTPS client' — Match the URL Scheme to the Endpoint
Fix Loki 'http: server gave HTTP response to HTTPS client': a scheme mismatch where https:// hits a plain-HTTP endpoint. Align it or enable TLS.
-
'Ingestion rate limit exceeded' — Raise or Rebalance the Tenant Rate Limit
Fix Loki 'Ingestion rate limit exceeded' (429) errors: diagnose per-tenant rate vs burst limits, distributor rejections, and bursty senders, then raise limits, shard traffic, or add backpressure.
-
We encountered an internal error status code: 500' — Retry, Then Alert on Sustained Failures
Fix Loki's 'InternalError: We encountered an internal error, please try again status code: 500': enable retries/backoff and alert only on sustained failures.
-
'invalid magic number' — Find and Remove the Corrupt Chunk or Index File
Fix Loki 'invalid magic number': a truncated or corrupt chunk or boltdb/TSDB index file. Remove the bad object and re-download clean.
-
'invalid schema config' — Set 24h Index Periods and Add New Schemas Safely
Fix Loki 'invalid schema config: boltdb-shipper works best with 24h periodic index config': set index period to 24h and align store/schema v13.
-
'invalid tenant ID' — Sanitize the X-Scope-OrgID Value
Fix Loki 'invalid tenant ID': tenant names with path separators, pipes, or over-length values are rejected. Map to safe tenant slugs.
-
'joining memberlist cluster: failed to reach any nodes' — Reconnect the Gossip Cluster
Fix Loki 'joining memberlist cluster: failed to reach any nodes': fix join_members, the headless service, and gossip port 7946.
-
'label value too long' — Keep High-Entropy Strings Out of Your Stream Labels
Fix Loki's 'label value too long': stop putting URLs, UUIDs and messages into labels, move them into the log body, and tune max_label_value_length safely.
-
'max concurrent tail requests limit exceeded' — Manage Live Tail Connections
Fix Loki's 'max concurrent tail requests limit exceeded': close stale live-tail sessions, raise the cap per tenant, or use bounded range queries.
-
'max entries limit per query exceeded' — Cap Results, Aggregate, and Paginate
Fix Loki's 'max entries limit per query exceeded': lower the requested line count, switch to metric queries, or paginate large log pulls.
-
'max line size exceeded' — Truncate, Split, or Raise max_line_size
Fix Loki 'max line size exceeded' rejections: find oversized log lines from stack traces and JSON blobs, then truncate at the source, split lines, or raise max_line_size safely per tenant.
-
'Maximum active stream limit exceeded' — Cut Label Cardinality and Raise Stream Limits
Fix Loki's 'Maximum active stream limit exceeded': drop high-cardinality labels, raise max_global_streams_per_user, add ingesters, and stop label explosion from filling active streams.
-
'maximum of series reached for a single query' — Reduce Cardinality or Raise the Limit
Fix Loki 'maximum of series reached for a single query': understand max_query_series, why high-cardinality LogQL explodes series, and how to narrow labels, aggregate, or safely raise the limit.
-
'mkdir /loki: permission denied' — Fix Volume Ownership and fsGroup
Fix Loki 'mkdir /loki: permission denied' at startup: set securityContext fsGroup for the non-root uid and use writable WAL/compactor mounts.
-
'no org id' — Supply the X-Scope-OrgID Tenant Header
Fix Loki 'no org id' errors: understand multi-tenancy and the X-Scope-OrgID header, why pushes and queries fail without a tenant, and how to set it in clients, Grafana, and the ruler.
-
'no space left on device' on the WAL — Stop the Ingester Disk From Filling Up
Fix Loki's WAL 'no space left on device': grow the WAL volume, fix the flush failures that stall segment truncation, and tune ingester.wal before disk fills.
-
'NoCredentialProviders: no valid providers in chain' — Give Loki Working Object-Store Credentials
Fix Loki 'NoCredentialProviders: no valid providers in chain' CredentialsError: bind IRSA/Workload Identity or set access_key_id to authenticate.
-
'NoSuchBucket: The specified bucket does not exist' — Point Loki at a Bucket That Exists
Fix Loki's 'NoSuchBucket: The specified bucket does not exist status code: 404': correct bucketnames, region, endpoint, and path-style so flushes land.
-
syntax error: unexpected IDENTIFIER' — Fix Invalid LogQL
Fix Loki LogQL 'parse error: syntax error: unexpected IDENTIFIER': understand stream selectors, filter and pipeline syntax, and how to correct matchers, quoting, and metric-query structure.
-
'Per stream rate limit exceeded' — Raise Stream Limits and Fix Label Cardinality
Fix Loki's 'Per stream rate limit exceeded': tune per_stream_rate_limit and burst, split hot streams with better labels, and stop one chatty stream from dropping logs at ingest.
-
'pipeline error' (__error__="JSONParserErr") — Handle Lines That Don't Match Your Parser
Fix Loki 'pipeline error' with __error__="JSONParserErr" or "LogfmtParserErr": filter failed lines and scope the parser to matching streams.
-
'queries require at least one regexp or equality matcher' — Fix Empty Stream Selectors
Fix Loki's 'queries require at least one regexp or equality matcher': add a concrete label matcher to the stream selector, avoid empty-compatible matchers, and write valid LogQL.
-
'query too large to execute on a single querier' — Narrow the Scan Before You Raise the Limit
Fix Loki's 'query too large to execute on a single querier: would read too many bytes': add stream selectors, shorten the range, enable sharding, tune limits.
-
'RequestTimeTooSkewed status code: 403' — Fix Clock Drift, Not IAM
Fix Loki 'RequestTimeTooSkewed ... status code: 403' storage errors by syncing the node clock with NTP, not by changing IAM.
-
'ResourceExhausted: grpc: received message larger than max' — Raise gRPC Message Size Limits
Fix Loki's 'ResourceExhausted ... message larger than max': raise grpc_server_max_recv_msg_size and client send limits, shrink batches and query fan-out, and stop oversized gRPC frames.
-
code = Unavailable desc = transport is closing' — Stabilize gRPC Between Components
Fix Loki 'rpc error: code = Unavailable desc = transport is closing': trace pod restarts and OOMKills, and tune gRPC keepalive.
-
'SlowDown: Please reduce your request rate status code: 503' — Throttle and Cache the S3 Path
Fix Loki's 'SlowDown: Please reduce your request rate status code: 503': add retries, chunk and index caches, and write fewer, larger chunks to S3.
-
'the query time range exceeds the limit' — Split the Range or Raise max_query_length
Fix Loki 'the query time range exceeds the limit': understand max_query_length and max_query_lookback, why long ranges are rejected, and how to split, shorten, or safely raise the limits.
-
'timestamp too new' — Fix Clock Skew Before the Distributor Rejects Future-Dated Logs
Fix Loki's 'entry for stream has timestamp too new': correct sender clocks and timestamp parsing, then tune creation_grace_period for legitimate future skew.
-
'too many chunks to fetch' — Narrow the Query or Raise max_chunks_per_query
Fix Loki 'too many chunks to fetch': understand max_chunks_per_query, why broad matchers and long ranges pull too many chunks, and how to tighten selectors, split ranges, or raise the limit safely.
-
'too many open files' — Raise the File-Descriptor Limit
Fix Loki 'accept4: too many open files': the OS fd ulimit is too low for open connections and chunk files. Raise LimitNOFILE and ulimits.
-
'too many outstanding requests' — Raise Querier Concurrency and Split Queries
Fix Loki's 'too many outstanding requests': tune max_outstanding_per_tenant, querier concurrency, query splitting and sharding, and the query-scheduler so heavy queries stop queueing up.
-
'too many unhealthy instances in the ring' — Restore Ring Health and Quorum
Fix Loki 'too many unhealthy instances in the ring': diagnose KV store heartbeat failures, unregistered ingesters, and lost quorum, then flush the ring, fix the KV backend, and restore replication.
-
'x509: certificate signed by unknown authority' — Trust the CA in the Client tls_config
Fix Loki 'x509: certificate signed by unknown authority': add the private CA to the client tls_config ca_file and include intermediates.
Loki AI prompts
Copy-paste, production-safe prompts for this stack.
More Loki guides
Every Loki prompt and troubleshooting guide.
Frequently asked questions
Why did Filebeat stop shipping logs without an obvious error?
harvester and registry lines and confirm the output is reachable. See too many open files.How do I find which grok pattern is breaking a Logstash pipeline?
_grokparsefailure and can stall the pipeline; enable --config.debug or test the pattern in isolation, and check for a stuck output blocking the pipeline queue. See Logstash instance lock for the related single-instance failure mode.