🚨 Redis Error Guide: 'Connection reset by peer' / Broken Pipe to Redis

Overview

Connection reset by peer (and its sibling Broken pipe) is a TCP-level error the client sees when Redis — or something in the network path — closes a socket the client still thinks is open. Redis itself did not necessarily crash; it often closes a connection deliberately (idle timeout, a tripped client output buffer limit, CLIENT KILL), or an intermediary (load balancer, NAT, firewall) drops an idle flow. To the client library this surfaces as a reset, EOF, or broken pipe on the next read/write.

The literal errors seen client-side:

redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

Error: write EPIPE   (broken pipe)

And on the Redis side, the corresponding log line when it deliberately closes a client:

Client ... scheduled to be closed ASAP for overcoming of output buffer limits.

Because “reset by peer” is generic, the job is to figure out why the peer (Redis or a middlebox) closed the connection.

Symptoms

• Client logs Connection reset by peer / EPIPE / broken pipe, often after an idle period or a large reply.
• Errors spike during big KEYS/SMEMBERS/HGETALL/pub-sub fan-out (large replies).
• Long-lived idle connections fail on first reuse (through a load balancer/NAT).
• Redis log shows clients closed for output buffer limits, or the OS OOM-killed redis-server.

redis-cli INFO clients | grep -E 'connected_clients|client_recent_max_output_buffer|blocked'

connected_clients:512
client_recent_max_output_buffer:41943040

Common Root Causes

1. Client output buffer limit exceeded

A client (especially a pub/sub subscriber or replica) that cannot drain data fast enough hits client-output-buffer-limit, and Redis force-closes it.

redis-cli CONFIG GET client-output-buffer-limit

# class  hard-limit soft-limit soft-seconds
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

A slow pub/sub consumer receiving a firehose blows past 32mb and is dropped.

2. Idle connection reaped by timeout or a middlebox

timeout N closes idle client connections; load balancers and NAT gateways also silently drop idle TCP flows, so the next use resets.

redis-cli CONFIG GET timeout
redis-cli CONFIG GET tcp-keepalive

3. Redis crashed / OOM-killed / restarted

If redis-server died, all connections reset at once.

dmesg | grep -i 'killed process' | grep -i redis
journalctl -u redis --no-pager | tail -20

Out of memory: Killed process 8123 (redis-server)

4. maxmemory with a blocking eviction / big reply

Under memory pressure or a huge single reply, a client’s buffer grows until the limit trips and Redis closes it.

Diagnostic Workflow

Step 1: Determine whether Redis stayed up

redis-cli PING
redis-cli INFO server | grep -E 'uptime_in_seconds|process_id'
journalctl -u redis --no-pager | tail -20

If uptime_in_seconds is small, Redis restarted (check dmesg for OOM). If it is large, Redis stayed up and closed individual connections.

Step 2: Check for output-buffer kills

journalctl -u redis --no-pager | grep -i 'output buffer' | tail
redis-cli INFO clients | grep client_recent_max_output_buffer
redis-cli CONFIG GET client-output-buffer-limit

A high client_recent_max_output_buffer plus “scheduled to be closed” log lines confirm buffer-limit resets.

Step 3: Check idle-reaping and keepalive settings

redis-cli CONFIG GET timeout
redis-cli CONFIG GET tcp-keepalive
redis-cli CLIENT LIST | grep -oE 'idle=[0-9]+' | sort -t= -k2 -n | tail

Long idle values plus a nonzero timeout (or a LB idle timeout) explain resets on reuse.

Step 4: Rule out memory pressure / OOM

redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'
dmesg | grep -i 'killed process'
cat /proc/sys/vm/overcommit_memory   # should be 1 for Redis

Example Root Cause Analysis

A metrics service using Redis pub/sub started logging Connection reset by peer every few minutes, always on its subscriber connection. Redis PING responded fine and uptime_in_seconds was large — Redis had not crashed.

journalctl -u redis --no-pager | grep -i 'output buffer' | tail -3

Client id=8821 addr=10.0.4.9:5510 ... scheduled to be closed ASAP for overcoming of output buffer limits.

The subscriber consumed messages slowly (blocking on downstream I/O) while the publisher fanned out a burst. Its pub/sub output buffer exceeded the 32mb hard limit and Redis dropped it — surfacing as Connection reset by peer to the client.

Two fixes: (1) make the consumer drain messages promptly (offload downstream work to a queue so it never blocks the socket), and (2) raise the pubsub buffer soft limit modestly for burst tolerance. The consumer change alone resolved it:

redis-cli CONFIG SET client-output-buffer-limit "pubsub 64mb 16mb 60"
redis-cli INFO clients | grep client_recent_max_output_buffer

After the consumer stopped blocking, client_recent_max_output_buffer stayed low and the resets stopped.

Prevention Best Practices

• Keep pub/sub and replica consumers fast; never block on downstream work while holding the Redis socket. Size client-output-buffer-limit for real bursts.
• Avoid huge single replies (KEYS, unbounded SMEMBERS/HGETALL); use SCAN/HSCAN/SSCAN with a cursor.
• Enable tcp-keepalive (e.g. 300) and set client socket keepalive so idle flows through LBs/NAT stay alive or fail fast.
• Align app pool idle timeouts with Redis timeout and any load-balancer idle timeout so reuse never hits a half-closed socket.
• Set vm.overcommit_memory=1 and a sane maxmemory + eviction policy so Redis is not OOM-killed under pressure.
• Use connection pools with retry-on-reset; see more Redis error guides.

Quick Command Reference

# Did Redis stay up?
redis-cli PING
redis-cli INFO server | grep uptime_in_seconds
journalctl -u redis | tail -20

# Output-buffer kills
journalctl -u redis | grep -i 'output buffer' | tail
redis-cli INFO clients | grep client_recent_max_output_buffer
redis-cli CONFIG GET client-output-buffer-limit

# Idle reaping / keepalive
redis-cli CONFIG GET timeout
redis-cli CONFIG GET tcp-keepalive
redis-cli CLIENT LIST | grep -oE 'idle=[0-9]+' | sort -t= -k2 -n | tail

# Memory / OOM
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human'
dmesg | grep -i 'killed process'

Conclusion

Connection reset by peer / broken pipe means the peer closed a socket the client still held open — usually Redis closing it deliberately, or a middlebox dropping an idle flow. The common causes:

1. A slow consumer tripping client-output-buffer-limit (classic for pub/sub and replicas).
2. Idle connections reaped by Redis timeout or a load balancer/NAT idle timeout.
3. Redis crashing or being OOM-killed (all connections reset together).
4. Memory pressure or a huge single reply inflating a client buffer past its limit.

Check whether Redis stayed up (uptime_in_seconds, dmesg), then look for output-buffer kill log lines and idle/keepalive mismatches. Most fixes are on the client side — drain fast, avoid giant replies, and align idle timeouts.