🚨 Loki Error Guide: 'empty ring' / 'too many unhealthy instances in the ring' — Recover the Hash Ring

Overview

Loki distributes work across instances using a hash ring stored in a KV backend (memberlist, Consul, or etcd). When the ring has no usable members, or too many members are unhealthy to satisfy the replication factor, Loki fails requests with:

empty ring

and the closely related quorum failure:

too many unhealthy instances in the ring

Distributors write to ingesters chosen from the ring; queriers and rulers consult it too. empty ring means Loki looked up the ring and found zero healthy instances — usually a KV/membership problem. too many unhealthy instances in the ring means members exist but not enough are ACTIVE to meet replication_factor, so Loki cannot form a write/read quorum. Both take the affected path down until ring membership is restored.

Symptoms

• Pushes and queries fail cluster-wide with empty ring or too many unhealthy instances in the ring.
• The /ring status page shows instances as Unhealthy, LEAVING, or missing entirely.
• Errors spike right after a rollout, scale-down, or network partition between components.
• memberlist logs show members failing to gossip; Consul/etcd shows stale registrations.
• replication_factor (e.g. 3) exceeds the number of currently healthy ingesters.

Common Root Causes

• KV store connectivity broken — memberlist gossip blocked, or Consul/etcd unreachable, so no instances register.
• Stale ring entries — crashed ingesters never removed themselves, leaving Unhealthy tombstones.
• replication_factor higher than healthy ingester count — e.g. RF=3 with only 2 running.
• Misconfigured memberlist.join_members so pods never form a cluster.
• All ingesters restarting at once during a bad rollout, momentarily emptying the ring.
• Clock skew or firewall dropping the gossip/health-check port (default 7946 for memberlist).

Diagnostic Workflow

Inspect the ring directly — this is the fastest way to see membership and health:

# Ingester ring status as JSON
curl -s http://loki:3100/ring | head -40
# Also: /distributor/ring, /compactor/ring, /ruler/ring

Check ring/KV and replication settings in config:

common:
  replication_factor: 3
  ring:
    kvstore:
      store: memberlist
memberlist:
  join_members:
    - loki-memberlist:7946   # headless service for gossip
  bind_port: 7946
ingester:
  lifecycler:
    ring:
      replication_factor: 3

Verify gossip connectivity between instances:

# From one Loki pod, confirm the gossip port is reachable on a peer
nc -vz loki-1.loki-memberlist 7946
journalctl -u loki --since '10 min ago' | grep -i memberlist

Count healthy ingesters against replication_factor. If healthy < RF, that alone explains too many unhealthy instances in the ring.

Example Root Cause Analysis

A cluster ran three ingesters with replication_factor: 3 on memberlist. A node drain restarted two ingesters simultaneously. They crashed before deregistering, leaving two Unhealthy tombstones in the ring. With only one healthy ingester against RF=3, distributors could not form a quorum and every push failed with too many unhealthy instances in the ring; briefly, while the last ingester also cycled, queries returned empty ring.

The team confirmed the state on /ring, saw two stale Unhealthy entries, and used the ring page’s “Forget” action to evict the dead instances. As the restarted ingesters came back ACTIVE, quorum was restored and pushes resumed. To prevent recurrence they set a PodDisruptionBudget so no more than one ingester restarts at a time, and confirmed join_members pointed at the headless memberlist service so new pods always rejoined the gossip cluster cleanly.

Prevention Best Practices

• Keep healthy ingester count >= replication_factor at all times; never scale ingesters below RF.
• Use a PodDisruptionBudget so rollouts and drains restart ingesters one at a time.
• Point memberlist.join_members (or Consul/etcd) at a stable headless service so pods reliably rejoin.
• Ensure the gossip port (7946 for memberlist) is open between all Loki instances.
• Monitor the /ring endpoints and alert on any Unhealthy or LEAVING members.
• Use the ring “Forget” action (or cortex_ring_members metrics) to clear tombstones promptly after crashes.

Quick Command Reference

# Ring membership and health
curl -s http://loki:3100/ring

# Gossip reachability and memberlist logs
nc -vz <peer> 7946
journalctl -u loki | grep -i memberlist

# Confirm replication factor
curl -s http://loki:3100/config | grep replication_factor

common:
  replication_factor: 3
memberlist:
  join_members:
    - loki-memberlist:7946

Conclusion

empty ring and too many unhealthy instances in the ring are hash-ring health failures: either no instances registered (a KV/gossip problem) or too few are healthy to meet replication_factor. Diagnose from the /ring endpoints, clear stale Unhealthy tombstones with Forget, and confirm gossip connectivity and membership. Keep healthy ingesters at or above the replication factor and stagger restarts with a PodDisruptionBudget so the ring never loses quorum during normal operations.