🚨 Loki Error Guide: 'too many unhealthy instances in the ring' — Restore Ring Health and Quorum

Overview

Loki cannot satisfy the replication quorum because too many ring members are marked unhealthy. Writes and reads fail with:

too many unhealthy instances in the ring

The ring is the hash-ring stored in a KV backend (memberlist, Consul, or etcd) that tracks which ingesters own which streams. Each ingester heartbeats into the ring; when more than replication_factor - quorum members go Unhealthy (missed heartbeats), the distributor can no longer write to enough replicas and rejects the request.

Symptoms

• Push requests fail with too many unhealthy instances in the ring; queries may also fail or return partial data.
• The /ring admin page shows members in Unhealthy state with stale Last Heartbeat.
• Errors appear right after a rollout, scale-down, node failure, or network partition.
• loki_ring_members{state="Unhealthy"} is non-zero.
• Distributor logs show at least N live replicas required, could only find M.

Common Root Causes

• Ingesters crashed or OOMKilled — replicas died faster than they could deregister, leaving stale ring entries.
• KV store problems — memberlist gossip partition, or an unreachable/unhealthy Consul/etcd backend.
• Aggressive scale-down — too many ingesters removed at once, dropping below quorum before handoff completed.
• Clock skew or missed heartbeats — heartbeat_timeout exceeded due to CPU starvation or network latency.
• Unclean shutdowns — pods killed without graceful deregistration, leaving Unhealthy tombstones.
• Misconfigured replication_factor — RF higher than the number of running ingesters.

Diagnostic Workflow

Inspect the ring state directly (ingester ring endpoint):

kubectl port-forward svc/loki-ingester-headless 3100 &
curl -s http://localhost:3100/ring | grep -iE 'State|Heartbeat|Tokens' | head

Count members by state:

sum by (state) (loki_ring_members{name="ingester"})

Check the KV backend and heartbeat settings:

common:
  ring:
    kvstore:
      store: memberlist
    heartbeat_period: 5s
    heartbeat_timeout: 1m
    replication_factor: 3

Confirm the running ingester count meets the replication factor:

kubectl get pods -l app.kubernetes.io/component=ingester

If entries are stale Unhealthy tombstones for pods that no longer exist, forget them from the ring:

# From the ring admin page, "Forget" the dead instance, or via API:
curl -s -X POST "http://localhost:3100/ring" \
  --data-urlencode "forget=ingester-loki-2-<oldid>"

For memberlist partitions, verify gossip connectivity between pods:

kubectl logs -l app.kubernetes.io/component=ingester --since=10m \
  | grep -iE 'memberlist|gossip|failed to join'

Example Root Cause Analysis

After a node drain, a Loki cluster (RF=3, 3 ingesters) started rejecting all writes with too many unhealthy instances in the ring. The /ring page showed one live ingester, one JOINING, and one Unhealthy tombstone for a pod that had been evicted without graceful shutdown. With only one healthy owner for many token ranges and RF=3, quorum (2 live replicas) could not be met.

Two things were wrong. First, the evicted pod never deregistered, so its Unhealthy entry lingered past heartbeat_timeout. They used the ring’s “Forget” action to remove the dead instance immediately. Second, the replacement ingester was stuck JOINING because it was OOMKilled replaying its WAL; they raised its memory limit so it could finish WAL replay and reach ACTIVE. Once two ingesters were ACTIVE, quorum was restored and writes resumed. Long-term, they enabled a PodDisruptionBudget and graceful shutdown so future drains hand off tokens before a pod leaves.

Prevention Best Practices

• Set a PodDisruptionBudget on ingesters so node drains never remove more than one at a time.
• Ensure graceful shutdown (SIGTERM handling, -ingester.flush-on-shutdown-with-wal-enabled) so leaving ingesters deregister cleanly.
• Right-size ingester memory so WAL replay on restart doesn’t OOM and strand a member in JOINING.
• Keep the number of running ingesters comfortably above replication_factor.
• Use memberlist with enough peers, or a healthy dedicated Consul/etcd, and monitor KV backend health.
• Alert on loki_ring_members{state="Unhealthy"} > 0 so you catch tombstones before quorum is lost.

Quick Command Reference

# Ring state
curl -s http://localhost:3100/ring | grep -iE 'State|Heartbeat'

# Members by state
sum by (state) (loki_ring_members{name="ingester"})

# Forget a dead instance
curl -s -X POST "http://localhost:3100/ring" --data-urlencode "forget=<instance-id>"

# Memberlist/gossip health
kubectl logs -l app.kubernetes.io/component=ingester --since=10m | grep -iE 'memberlist|gossip'

Conclusion

too many unhealthy instances in the ring means Loki lost quorum because more ring members are Unhealthy than the replication factor tolerates — usually from crashed/OOMKilled ingesters, ungraceful shutdowns leaving tombstones, or a KV-store partition. Inspect /ring, forget stale dead instances, restore enough ACTIVE ingesters to meet quorum, and fix the underlying cause (memory, PDB, graceful shutdown, KV health). Add a PDB and alert on unhealthy members so ring quorum survives the next drain or node failure.