🚨 Logstash Error Guide: 'Logstash could not be started because there is already another instance' — Clear the data.lock

Overview

Logstash guards its data directory with a lock file so two processes cannot corrupt the same persistent queue and dead-letter queue. When it cannot acquire that lock, startup aborts immediately:

[FATAL][org.logstash.Logstash] Logstash could not be started because there is already
another instance using the configured data directory.
If you wish to run multiple instances, you must change the "path.data" setting.

The lock is the file path.data/.lock (default /var/lib/logstash/.lock or /usr/share/logstash/data/.lock). Either a second Logstash is genuinely running, or a previous instance died without releasing the lock and left it stale.

Symptoms

• Logstash exits within a second of starting; the log ends with the FATAL ... another instance line.
• systemctl restart logstash fails and the unit goes to failed or a restart loop.
• A crash-looping container restarts endlessly with the same message.
• Two pipelines or two systemd units were configured to share one path.data.
• After an ungraceful kill (OOM, kill -9, node reboot mid-write), Logstash refuses to start even though nothing else is running.

Common Root Causes

• A real second instance — a manual bin/logstash run while the systemd service is also up, or two units pointing at the same directory.
• Stale lock after a hard kill — kill -9, an OOM kill, or a power loss left .lock behind with no owner.
• Shared path.data across pipelines — multiple pipelines.yml entries or Docker containers mounting the same volume.
• A previous process still shutting down — a large persistent queue is flushing and the old JVM has not fully exited.
• Wrong path.data permissions — the file exists but the service user cannot remove or rewrite it after an unclean exit.

Diagnostic Workflow

First find out whether a Logstash is actually running before touching any lock:

pgrep -af 'org.logstash.Logstash'
ps -ef | grep -i '[l]ogstash'
systemctl status logstash --no-pager

Locate the effective path.data — it may be overridden on the command line or in logstash.yml:

grep -E '^\s*path.data' /etc/logstash/logstash.yml
# Default locations:
ls -la /var/lib/logstash/.lock /usr/share/logstash/data/.lock 2>/dev/null

Confirm whether the lock is stale by checking for any process holding the file open:

sudo lsof /var/lib/logstash/.lock 2>/dev/null
sudo fuser /var/lib/logstash/.lock 2>/dev/null

If two pipelines are colliding, inspect pipelines.yml for a per-pipeline path.data:

# /etc/logstash/pipelines.yml — each pipeline needs its own data dir if separated
- pipeline.id: apache
  path.config: "/etc/logstash/conf.d/apache.conf"
  path.data: "/var/lib/logstash/apache"
- pipeline.id: syslog
  path.config: "/etc/logstash/conf.d/syslog.conf"
  path.data: "/var/lib/logstash/syslog"

Only after confirming nothing holds the lock, remove it and restart:

sudo systemctl stop logstash
sudo lsof /var/lib/logstash/.lock 2>/dev/null || sudo rm -f /var/lib/logstash/.lock
sudo systemctl start logstash

If Logstash feeds Elasticsearch, verify the pipeline came back online end-to-end:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep pipeline
curl -s 'http://es:9200/_cat/indices/logstash-*?v'

Example Root Cause Analysis

An operator upgraded Logstash on a busy host. The old process had a 2 GB persistent queue and was mid-flush when the deploy script issued kill -9 after a short timeout. The JVM died before releasing /var/lib/logstash/.lock, so the new systemd unit crash-looped with another instance using the configured data directory.

pgrep -af org.logstash.Logstash returned nothing and lsof /var/lib/logstash/.lock showed no holder — proof the lock was stale, not live. The operator stopped the unit, removed the orphaned .lock, and restarted. Logstash replayed the persisted queue on boot and resumed. The deploy script was then fixed to send SIGTERM and wait for a clean exit (raising TimeoutStopSec) before ever escalating to SIGKILL, so future upgrades release the lock cleanly.

Prevention Best Practices

• Give every Logstash instance and every separated pipeline its own unique path.data; never share the directory across processes or containers.
• Stop Logstash with SIGTERM and allow a generous TimeoutStopSec (large persistent queues take time to flush) so the lock is released cleanly.
• In systemd, set Restart=on-failure with a small RestartSec but avoid tight loops that repeatedly hit the lock; fix the root cause instead.
• In Kubernetes/Docker, mount a distinct volume per replica (use a StatefulSet with per-pod PVCs) rather than a shared ReadWriteMany claim.
• After any hard kill or node crash, script a startup pre-check that verifies no process holds .lock before deleting it.
• Ensure the service user owns path.data so it can create and remove the lock without permission errors.

Quick Command Reference

# Is a Logstash actually running?
pgrep -af 'org.logstash.Logstash'
systemctl status logstash --no-pager

# Where is path.data / the lock?
grep -E '^\s*path.data' /etc/logstash/logstash.yml
ls -la /var/lib/logstash/.lock

# Who (if anyone) holds the lock?
sudo lsof /var/lib/logstash/.lock
sudo fuser /var/lib/logstash/.lock

# Clear a confirmed-stale lock and restart
sudo systemctl stop logstash
sudo rm -f /var/lib/logstash/.lock
sudo systemctl start logstash

# Verify recovery
curl -s localhost:9600/_node/stats/pipelines?pretty | grep pipeline

Conclusion

The another instance using the configured data directory error is Logstash protecting its queue from concurrent writers. Diagnose it by proving whether a process is truly running: if pgrep and lsof both come up empty, the .lock is stale and safe to remove after stopping the service. If something is running, you have a duplicate instance or two pipelines sharing one path.data — give each its own directory. Prevent recurrence by shutting Logstash down gracefully with SIGTERM and a long stop timeout so the lock is always released on exit.