📜 RabbitMQ Streams as a Replayable Log With AI

The first time I reached for RabbitMQ Streams, I did it for the wrong reason. We had a queue that several services consumed, and I wanted each service to read the full history independently. A Stream gave me that — and then someone asked why disk usage kept climbing in steps instead of smoothly, and I realized I understood the feature about half as well as I’d assumed. Streams are not queues with extra features. They’re an append-only, replayable log with offset-based reads and policy-based retention, and treating them like a durable queue is how you end up surprised by disk graphs.

This is a good place to lean on AI, because the Stream-versus-queue decision is a reasoning problem and the retention math has a sharp edge that’s easy to get wrong. The model is genuinely useful for talking through whether your workload wants a log at all — but you have to make it justify the choice, and you have to verify the retention behavior yourself, because it will happily quote max-length-bytes as if that’s the whole story.

Make the AI argue Stream vs queue first

Before any config, the real question is whether you need a log or a queue. A queue deletes messages on ack; a Stream keeps them and lets consumers read by offset. If your consumers read once and never replay, you probably want a quorum queue with less operational surface.

I have four services that each need to read the full history of order events independently, and a new service occasionally needs to replay from the beginning. Throughput is moderate. Should I use a RabbitMQ Stream or a quorum queue with multiple consumers? Make the case for each, then tell me the deciding factor for my situation.

The answer you want names independent offset-based replay as the thing only a Stream gives you cleanly. If the model just says “use a Stream because it’s newer,” push back. The deciding factor here is genuine replay from arbitrary offsets by independent readers — without that, the simpler queue wins.

Declare the Stream and set retention deliberately

A Stream is declared as a queue with x-queue-type: stream, but the settings that matter are retention and segment size.

# Declare a stream with retention via rabbitmqadmin
rabbitmqadmin declare queue name=order-events \
  queue_type=stream \
  arguments='{"x-max-length-bytes": 21474836480, "x-stream-max-segment-size-bytes": 524288000}'

Or apply retention with a policy so it’s centrally managed:

rabbitmqctl set_policy stream-retention "^order-events$" \
  '{"max-age": "7D", "stream-max-segment-size-bytes": 524288000}' \
  --apply-to queues

Here’s the edge that bit me: retention deletes whole segments, not individual messages. A Stream truncates only when an entire segment ages or sizes out, so effective retention overshoots your nominal limit, and disk usage moves in lumps the size of a segment. If your segment is 500 MB and your retention is “roughly 20 GB,” you’ll routinely sit above 20 GB until the next full segment rolls off. Size disk headroom against segment size, not just the limit.

Offsets are where consumers surprise you

The other half of Streams is that consumers read by offset and are responsible for remembering where they were. A consumer that never commits its offset re-reads the entire log on every restart.

# Inspect a stream queue's depth and committed offsets via the stream tools
rabbitmqctl list_queues name type messages --vhost / | grep stream

When a new consumer attaches, it chooses a starting point — first, last, next, or a specific offset/timestamp. Production incidents come from consumers that default to first and re-process gigabytes on every deploy, or from offset tracking that was never wired up at all. Decide the offset strategy explicitly, and watch committed-offset lag per consumer the way you’d watch queue depth on a classic queue.

Where AI overreaches on Streams

The model tends to oversell Streams as a Kafka replacement and gloss over the operational differences. Streams have their own log replication and a stream coordinator; they are not classic mirrored queues, and the replication factor has real disk and network cost at your throughput. When the AI hands you a config, ask it to show the disk math: message size times rate times retention window times replication factor. If it can’t, the retention values are guesses.

It also tends to forget the segment-truncation behavior the moment the conversation moves on. I keep a single verification question in my back pocket: “Given a 500 MB segment size and a 20 GB max-length-bytes, what’s my realistic worst-case disk usage for this stream?” A good answer accounts for the overshoot; a bad one repeats 20 GB and proves it forgot.

My loop for adopting Streams

I make the AI win the Stream-versus-queue argument first, then have it produce retention and segment settings with the disk math shown, then I stand the Stream up on a staging cluster with representative message sizes and drive synthetic load. I watch disk usage climb in segment-sized steps to confirm the overshoot matches the math, and I verify a fresh consumer commits offsets instead of re-reading from first. The AI drafts the reasoning and the config; the staging cluster tells me whether the disk and offset behavior is what I actually signed up for.

If you’re weighing Streams against quorum queues, the quorum vs classic mirrored queues guide covers the queue side of that decision, and the broader RabbitMQ category collects the routing and retention topics Streams have to coexist with. The reasoning prompts I use for this live with my other prompts.