🐰 RabbitMQ Error: Stream Publisher Confirms Timing Out

Exact Error Message

A stream publisher waiting for confirmations that never come times out in the client, while the broker log shows the underlying stress:

io.rabbitmq.stream.StreamException: Confirm timeout after 30000 ms
    for 1024 outstanding messages on publisher 'ingest-1' (stream 'events')

2026-07-17 12:41:09.517 [warning] <0.1330.0> osiris_writer for stream 'events'
    on rabbit@mq-02: write latency high, last batch fsync 4180 ms
2026-07-17 12:41:12.004 [warning] <0.1188.0> memory resource limit alarm set on
    node rabbit@mq-02, publishers will be blocked

The client-side symptom is an ever-growing count of unconfirmed messages: the publisher keeps sending, but confirm callbacks stop firing and eventually the client raises a confirm timeout.

What It Means

Stream publishers use the RabbitMQ stream protocol with publisher confirms: each published message carries a publishing id, and the broker sends a confirm back once the message is durably written to the stream’s on-disk log (backed by osiris). A confirm timeout means the broker has not acknowledged those messages within the client’s confirm window.

Confirms are gated on the write actually being persisted (and, for a replicated stream, acknowledged by enough members). So a confirm timeout almost always points at write-path pressure: a resource alarm blocking the connection, slow disk fsync on the osiris log, or a stream that can’t reach its required replication because members are down.

Common Causes

• A memory or disk resource alarm on the stream’s leader node, which blocks publishing connections until the alarm clears.
• Slow or saturated disk under the osiris segment directory, so fsync latency balloons and confirms lag.
• The stream is under-replicated — one or more members are down, so writes wait on replication acks that never arrive.
• The client’s confirm timeout / outstanding-confirms window is too small for the current throughput and batch size.
• A leader election in progress after the stream leader restarted, briefly stalling writes.
• Network latency between stream members inflating the time to reach the required replication.

Diagnostic Commands

Check for resource alarms first — a set alarm blocks all publishers:

rabbitmq-diagnostics alarms

Inspect the stream’s members and which node leads it:

rabbitmq-streams stream_status events

Look at memory and disk-free headroom on the leader node:

rabbitmq-diagnostics memory_breakdown -n rabbit@mq-02
rabbitmq-diagnostics check_alarms
df -h /var/lib/rabbitmq

Watch the log for osiris write-latency and alarm lines while publishing:

sudo journalctl -u rabbitmq-server -f | grep -iE 'osiris|alarm|stream'

Confirm the stream’s replication factor and current member health:

rabbitmqctl list_queues name type members online_members | grep -i stream

Step-by-Step Resolution

1. Check alarms immediately. If rabbitmq-diagnostics alarms shows a memory or disk alarm on the leader, that alone blocks publishers — no confirms will flow until it clears.

2. For a disk alarm, free space or lower the pressure, then confirm the free-disk watermark is satisfied:

df -h /var/lib/rabbitmq
rabbitmq-diagnostics check_alarms

Retention trims old segments, so tightening max-age/max-length-bytes on the stream reclaims disk:

rabbitmqctl set_policy stream-retention '^events$' \
  '{"max-length-bytes":21474836480}' --apply-to queues

3. For a memory alarm, identify what is consuming memory and relieve it; publishers unblock automatically once the alarm resets.

4. If no alarm is set but osiris logs high fsync latency, the disk is the bottleneck. Move /var/lib/rabbitmq to faster storage or reduce competing I/O; confirm latency drops in the log.

5. If the stream is under-replicated, restore the down members so writes stop waiting on missing replication:

rabbitmq-streams stream_status events
# bring the offline member's node back, then re-check online_members

6. If throughput is simply high and healthy, raise the client’s confirm timeout and outstanding-confirms window, and batch publishes so the broker isn’t fsync-bound per message.

7. Confirm the publisher recovers — outstanding confirms should drain to near zero:

rabbitmq-streams stream_status events

The committed_chunk_id should be advancing and the log should be free of osiris latency and alarm warnings.

Prevention

• Alert on rabbitmq-diagnostics alarms and on free-disk/memory watermarks so an alarm pages before publishers stall.
• Provision fast, dedicated disk for the osiris log; confirm latency is dominated by fsync, not CPU.
• Keep streams fully replicated and alert when online_members drops below the configured replication factor.
• Size retention so disk never approaches the free-disk watermark under peak ingest.
• Tune the client’s confirm window and batch size to real throughput rather than defaults, and treat confirm timeouts as backpressure to slow the publisher.

For prompts that turn an alarm and stream_status snapshot into a tuning plan, see the RabbitMQ prompt library.

Related Errors

• connection.blocked / resource alarm — the alarm that blocks publishers and stalls confirms.
• Stream offset out of range — the consumer-side stream error.
• precondition_failed — declaring a stream with a replication factor its cluster can’t satisfy.
• basic.nack — the classic-queue confirm-failure signal, distinct from a stream confirm timeout.

Frequently Asked Questions

Why did all confirms stop at once? That pattern almost always means a resource alarm was set on the leader node. RabbitMQ blocks publishing connections while a memory or disk alarm is active, so every in-flight confirm stalls until the alarm clears.

Is a confirm timeout a client bug or a broker problem? Usually the broker’s write path — slow fsync, an alarm, or under-replication — is slower than the client’s confirm window. Fix the broker-side pressure first; only raise the client timeout once the disk and replication are healthy.

How do I know if the disk is the bottleneck? Look for osiris write-latency warnings in the log and check fsync times. If confirms lag while df shows healthy space but latency is high, move the osiris log to faster storage.

Does raising the client confirm timeout fix it? Only if the delay is genuine, healthy backpressure. If an alarm or a down member is the cause, a longer timeout just delays the same failure — resolve the resource or replication issue instead.

How does retention relate to confirm timeouts? If retention is too loose, the stream’s disk usage can trip the free-disk alarm, which blocks publishers. Tightening max-age/max-length-bytes reclaims space and prevents the alarm. For more streaming patterns, see the RabbitMQ guides.