🚨 Loki Error Guide: 'at least 1 live replicas required, could only find 0' — Restore Ingester Write Quorum

Overview

The distributor rejects a push with HTTP 500 when it cannot reach enough healthy ingesters to satisfy the write quorum implied by replication_factor:

at least 1 live replicas required, could only find 0 - unhealthy instances: 10.0.1.5:9095,10.0.1.6:9095

The distributor hashes each stream to a set of ingesters in the ring and needs a minimum number of them ACTIVE to accept the write. Here it found zero live replicas and named the ingesters it tried — reachable in the ring by name but not usable, either because they are down or because their gRPC endpoint on port 9095 is unreachable. This is distinct from the empty ring page state: the ring is not empty, it lists members, but those members cannot take the write. Every push fails until enough ingesters return to ACTIVE.

Symptoms

• Distributor logs at least 1 live replicas required, could only find 0 and client pushes get HTTP 500.
• The named unhealthy instances in the message are ingester pod IPs on gRPC port 9095.
• /ring shows ingesters as Unhealthy, LEAVING, or with a stale heartbeat timestamp.
• Errors begin right after a rollout, node drain, or a wave of ingester OOMKills.
• Queries may still partially work from flushed chunks while writes fail hard.

Common Root Causes

• Ingesters crashed, OOMKilled, or rolling all at once — no ingester is ACTIVE, so the distributor finds zero live replicas.
• gRPC port 9095 blocked by a NetworkPolicy — ingesters are running but the distributor cannot open a gRPC connection to them, so they count as unhealthy.
• replication_factor higher than the number of healthy ingesters — the quorum math cannot be satisfied even though some ingesters are up.
• Stale unhealthy entries in the ring — crashed ingesters left tombstones that never deregistered, poisoning the replica set.
• Readiness/heartbeat lag — ingesters restarted and have not yet re-registered a fresh heartbeat, so the ring still considers them unhealthy.

How to diagnose

1. Read the ring directly — see how many ingesters are ACTIVE versus Unhealthy and confirm the IPs from the error message:

   curl -s http://loki-distributor:3100/ring | head -40
   # ingester ring; look for state ACTIVE vs Unhealthy and heartbeat age

2. Check ingester pod health and restarts — confirm whether ingesters crashed or were OOMKilled:

   kubectl get pods -l app=loki,component=ingester
   kubectl describe pod loki-ingester-0 | grep -A5 'Last State\|Reason'

3. Test the gRPC path from the distributor — verify port 9095 is actually reachable to the named ingester IP:

   kubectl exec -it deploy/loki-distributor -- nc -vz 10.0.1.5 9095

4. Compare healthy count against replication_factor — if healthy ingesters are fewer than RF, that alone explains the failure:

   common:
     replication_factor: 3
   ingester:
     lifecycler:
       ring:
         replication_factor: 3

5. Inspect NetworkPolicies on the ingester namespace — confirm nothing is dropping ingress on 9095:

   kubectl get networkpolicy -o yaml | grep -A8 '9095\|ingester'

Fixes

Restore or scale ingesters to at least replication_factor so the distributor can form a quorum. Bring crashed ingesters back and confirm they reach ACTIVE on /ring:

kubectl rollout status statefulset/loki-ingester
kubectl scale statefulset/loki-ingester --replicas=3   # >= replication_factor

Stagger restarts with a PodDisruptionBudget so a rollout or node drain never takes all ingesters down at once, which is what emptied the replica set:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-ingester-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: loki
      component: ingester

Forget the dead instances on the ring so stale Unhealthy tombstones stop counting against the replica set. Use the /ring page Forget action, or the API:

curl -s -X POST http://loki-distributor:3100/ingester/ring \
  -d 'forget=10.0.1.5:9095'
# or click "Forget" next to the dead instance on the /ring status page

Open gRPC 9095 between distributor and ingesters so running ingesters stop being counted as unreachable. Allow the port explicitly in the NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: loki-ingester-grpc
spec:
  podSelector:
    matchLabels: { app: loki, component: ingester }
  ingress:
    - from:
        - podSelector: { matchLabels: { app: loki, component: distributor } }
      ports:
        - protocol: TCP
          port: 9095

Lower replication_factor only as a deliberate trade-off if you genuinely run fewer ingesters than RF and cannot scale up. Reducing RF reduces write durability, so treat it as a last resort, not a routine fix:

common:
  replication_factor: 1   # single-ingester dev/test only; loses redundancy

What to watch out for

• This is not the same as empty ring: here the ring has members but none can take the write. Do not chase a KV/gossip problem when the ring is populated but the ingesters are down or unreachable.
• Keep healthy ingesters at or above replication_factor at all times; scaling ingesters below RF guarantees this error during any disruption.
• A NetworkPolicy that blocks gRPC 9095 makes healthy, running ingesters look dead — the pods pass their own readiness checks while the distributor still cannot write to them.
• Clear tombstones promptly; a crashed ingester that never deregistered keeps counting against the replica set long after the pod is gone.
• After a mass restart, give ingesters time to re-register a fresh heartbeat before assuming a deeper fault — brief windows of this error during a controlled rollout can be normal.

Related

• Loki Error Guide: ‘empty ring’ — the related ring-membership failure where no instances register at all.
• Loki Error Guide: ‘too many unhealthy instances in the ring’ — the quorum shortfall when members exist but too few are healthy.
• Loki Error Guide: ‘failed to flush chunks’ — what an unhealthy ingester hits next on the write path.