🚨 Loki Error Guide: 'joining memberlist cluster: failed to reach any nodes' — Reconnect the Gossip Cluster

Overview

When Loki uses memberlist for its hash ring, every instance must gossip with its peers over port 7946. If a starting pod cannot find any peers to join, it logs:

level=warn msg="joining memberlist cluster: failed to reach any nodes" ... "Failed to resolve loki-memberlist:7946"

and, once running alone, reports:

no memberlist members

The instance tried to resolve and dial the addresses in memberlist.join_members, failed to reach any of them, and could not form or join the gossip cluster. Without a gossip cluster there is no shared ring, so the ring stays empty or single-member and the read/write path degrades. The cause is almost always DNS, the headless service, or the gossip port — not Loki itself. The fix is to make join_members resolve to real peer endpoints on an open port 7946.

Symptoms

• Startup logs show failed to reach any nodes and Failed to resolve loki-memberlist:7946.
• Instances run in isolation and log no memberlist members.
• /ring shows only the local instance, or an empty/single-member ring.
• The loki-memberlist headless service has no endpoints, or the DNS name does not resolve.
• Errors appear on fresh pods that started before DNS or peers were ready.

Common Root Causes

• Wrong or absent memberlist.join_members — the list points at a name that does not exist, so no peers are ever contacted.
• Headless memberlist Service missing or has no endpoints — the loki-memberlist service is undefined, has the wrong selector, or matches no ready pods.
• Gossip port 7946 blocked — a NetworkPolicy or firewall drops TCP/UDP 7946, so peers cannot gossip even when DNS resolves.
• Wrong bind_addr/advertise_addr — the instance advertises an address peers cannot reach, so joins fail asymmetrically.
• Pods started before DNS was ready — the headless service records had not propagated when the pod first tried to join.

How to diagnose

1. Resolve the memberlist name from inside a pod — confirm the join target actually resolves to peer IPs:

   kubectl exec -it loki-ingester-0 -- nslookup loki-memberlist
   # expect one A record per ready Loki pod

2. Check the headless service and its endpoints — a service with no endpoints resolves to nothing:

   kubectl get svc loki-memberlist -o wide
   kubectl get endpoints loki-memberlist

3. Test the gossip port to a peer — verify TCP 7946 is reachable:

   kubectl exec -it loki-ingester-0 -- nc -vz loki-memberlist 7946

4. Review the memberlist config — confirm join_members and bind_port are set correctly:

   memberlist:
     join_members:
       - loki-memberlist:7946
     bind_port: 7946
   common:
     ring:
       kvstore:
         store: memberlist

5. Check the NetworkPolicy for 7946 — memberlist needs both TCP and UDP:

   kubectl get networkpolicy -o yaml | grep -A8 '7946\|memberlist'

Fixes

Point join_members at the headless memberlist service so every pod discovers its peers through stable DNS rather than hard-coded IPs:

memberlist:
  join_members:
    - loki-memberlist:7946   # headless service that lists all Loki pods
  bind_port: 7946

Ensure the headless service exists with the correct selector so its endpoints include every Loki pod that should gossip:

apiVersion: v1
kind: Service
metadata:
  name: loki-memberlist
spec:
  clusterIP: None            # headless: DNS returns each pod IP
  selector:
    app: loki                # must match the Loki pod labels
  ports:
    - name: gossip
      port: 7946
      protocol: TCP

Open TCP and UDP 7946 between all Loki pods so gossip and health checks are not dropped. Memberlist uses both protocols:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: loki-memberlist-gossip
spec:
  podSelector:
    matchLabels: { app: loki }
  ingress:
    - from:
        - podSelector: { matchLabels: { app: loki } }
      ports:
        - { protocol: TCP, port: 7946 }
        - { protocol: UDP, port: 7946 }

Set the advertise address explicitly when pods advertise an address peers cannot reach — bind to the pod IP so joins are symmetric:

memberlist:
  bind_addr:
    - 0.0.0.0
  advertise_addr: ${POD_IP}   # inject via the downward API
  bind_port: 7946

Allow retry on join so a pod that started before DNS was ready recovers instead of giving up. Enable rejoin and keep the abort-if-join-fails behavior off:

memberlist:
  join_members:
    - loki-memberlist:7946
  min_join_backoff: 1s
  max_join_backoff: 1m
  max_join_retries: 10
  abort_if_cluster_join_fails: false

What to watch out for

• join_members must resolve to real endpoints; a name with no backing service silently yields a single-member ring and this warning on every start.
• The headless service selector must match the Loki pod labels exactly — a typo produces a service with zero endpoints that resolves to nothing.
• Memberlist needs both TCP and UDP on 7946; opening only TCP leaves health checks failing and members flapping.
• A pod that starts before DNS is ready will log this once and should recover if retry/rejoin is enabled — do not treat a single startup warning as a persistent fault.
• advertise_addr mismatches cause asymmetric joins where a pod can dial peers but they cannot dial it back, leaving it half-joined.

Related

• Loki Error Guide: ‘empty ring’ — the ring failure that results when memberlist never forms a cluster.
• Loki Error Guide: ‘too many unhealthy instances in the ring’ — flapping gossip leaves members unhealthy and short of quorum.
• Loki Error Guide: ‘context deadline exceeded’ — timeouts that follow when components cannot reach each other over the ring.