🚨 Loki Error Guide: 'error getting ring ... connection refused' — Fix the KV Store Backend

Overview

When Loki’s hash ring is backed by an external KV store (Consul or etcd), every ring read goes to that backend. If the backend is unreachable, Loki logs:

level=error msg="error getting ring" err="Get \"http://consul:8500/v1/kv/collectors/ring\": dial tcp 10.0.0.9:8500: connect: connection refused"

The connection refused means the TCP dial to the KV backend reached a host but nothing was listening on the port — Consul on 8500 (or etcd on 2379) is down, the wrong address is configured, or a network policy blocks the port. Because Loki cannot read ring membership from the KV store, distributors, ingesters, queriers, and rulers all lose their view of the ring and the read/write path fails. The fix is to make the configured KV backend healthy and reachable — or to drop the external dependency by moving the ring to memberlist.

Symptoms

• Component logs repeat error getting ring with connection refused to consul:8500 or an etcd endpoint on 2379.
• The failure is cluster-wide and started when Consul/etcd was restarted, moved, or lost.
• /ring fails to render or shows stale membership.
• common.ring.kvstore.store is consul or etcd, not memberlist.
• Curling the KV backend directly from a Loki pod also fails to connect.

Common Root Causes

• Consul or etcd is down — the backend process crashed or was never deployed, so nothing listens on 8500/2379.
• Wrong KV address in common.ring.kvstore — kvstore.consul.host or kvstore.etcd.endpoints points at a name/IP that is not the live backend.
• NetworkPolicy blocking the KV port — 8500 (Consul) or 2379 (etcd) is dropped between Loki and the backend.
• TLS or ACL misconfiguration — the backend requires TLS or a valid ACL token that Loki is not sending, so connections are refused or rejected.
• store: consul with no cluster deployed — the config assumes an external KV backend that was never provisioned.

How to diagnose

1. Curl the KV backend from a Loki pod — reproduce the exact dial the error shows:

   kubectl exec -it deploy/loki-distributor -- \
     curl -s http://consul:8500/v1/status/leader
   # etcd variant: curl -s http://etcd:2379/health

2. Check the KV backend’s own health — confirm the process is up and serving:

   kubectl get pods -l app=consul
   kubectl logs -l app=consul --tail=50 | grep -i 'error\|leader'

3. Verify the configured address matches the running backend:

   common:
     ring:
       kvstore:
         store: consul
         consul:
           host: consul:8500     # must match the live Consul service

4. Test the port from a Loki pod — isolate a network-policy block from a dead backend:

   kubectl exec -it deploy/loki-distributor -- nc -vz consul 8500

5. Check for TLS/ACL requirements — a backend expecting TLS or a token refuses plain requests:

   kubectl get networkpolicy -o yaml | grep -A6 '8500\|2379'
   # confirm whether Consul ACLs are enabled and a token is required

Fixes

Restore the KV backend and confirm the configured address so Loki can read the ring again. Verify Consul/etcd is healthy and that the host in config resolves to it:

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul:8500
        http_client_timeout: 20s
        consistent_reads: true

Open the KV port between Loki and the backend so the dial is not refused by a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: loki-to-consul
spec:
  podSelector:
    matchLabels: { app: consul }
  ingress:
    - from:
        - podSelector: { matchLabels: { app: loki } }
      ports:
        - { protocol: TCP, port: 8500 }

Supply the correct ACL token (and TLS) the backend requires so connections are accepted rather than refused. Point Loki at the token file and CA:

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul:8500
        acl_token: ${CONSUL_HTTP_TOKEN}

For etcd, set the endpoints correctly so Loki dials the live cluster members on 2379:

common:
  ring:
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - etcd-0:2379
          - etcd-1:2379
        dial_timeout: 10s

Switch the ring to memberlist to remove the external dependency entirely if you do not need Consul/etcd. Gossip-based membership has no separate backend to fail:

common:
  ring:
    kvstore:
      store: memberlist
memberlist:
  join_members:
    - loki-memberlist:7946
  bind_port: 7946

What to watch out for

• connection refused means the port reached a host but nothing was listening — that is a dead backend or wrong address, not a DNS failure (which would say no such host).
• An external KV store is a hard dependency for the whole ring; if you keep Consul/etcd, run it highly available or Loki inherits its downtime.
• A NetworkPolicy block and a dead backend look identical from the log line; use nc -vz plus a direct backend health check to tell them apart.
• If the backend enforces ACLs or TLS, a missing token or CA can present as refused/rejected connections even when the process is healthy.
• Switching store to memberlist changes where ring state lives; migrate deliberately and confirm all components use the same store, or they will not share a ring.

Related

• Loki Error Guide: ‘empty ring’ — the ring failure that surfaces when the KV backend cannot be read.
• Loki Error Guide: ‘too many unhealthy instances in the ring’ — quorum loss when KV-backed membership goes stale.
• Loki Error Guide: ‘failed to flush chunks’ — a downstream write-path failure when ingesters lose their ring view.