🚨 Kafka Error Guide: 'OffsetOutOfRangeException' Fetch Position Out of Range

Exact Error Message

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Fetch position FetchPosition{offset=8421003, offsetEpoch=Optional[12], currentLeader=...} is out of range for partition payments-7
	at org.apache.kafka.clients.consumer.internals.Fetcher.initializeCompletedFetch(Fetcher.java:1455)
	at org.apache.kafka.clients.consumer.internals.Fetcher.collectFetch(Fetcher.java:687)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1271)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1230)
	at com.example.payments.PaymentConsumer.run(PaymentConsumer.java:71)

When auto.offset.reset is set, you will instead see an INFO/WARN that the offset was reset rather than an exception:

WARN  o.a.k.c.c.internals.Fetcher - [Consumer clientId=payment-worker-2, groupId=payments]
  Fetch position FetchPosition{offset=8421003, ...} is out of range for partition payments-7,
  resetting offset
INFO  o.a.k.c.c.internals.SubscriptionState - [Consumer clientId=payment-worker-2, groupId=payments]
  Resetting offset for partition payments-7 to position FetchPosition{offset=9105662, ...}.

What the Error Means

Every partition has a log start offset (the earliest offset still retained) and a log end offset (one past the last written record). A consumer’s fetch position must fall within that window. OffsetOutOfRangeException is thrown when the consumer asks the broker for an offset that is no longer (or not yet) in range — most commonly an offset that is below the log start offset because the underlying segment was deleted by retention before the consumer got to it.

This happens when a consumer is down or lagging long enough that retention (retention.ms or retention.bytes) garbage-collects the segments holding its next offset. When it resumes and tries to fetch offset 8,421,003 but the log now starts at 9,105,662, the position is invalid. What happens next depends on auto.offset.reset: earliest jumps to the new log start, latest jumps to the log end (skipping the gap), and none throws the exception to your code so you decide.

Common Causes

• Consumer lagged past retention. The group was stopped or slow, retention deleted the segments containing its committed offset, and the offset is now below the log start.
• auto.offset.reset=none with no fallback handling, so any out-of-range position surfaces as an exception instead of being reset.
• Retention shortened or retention.bytes hit. A topic config change or a traffic spike aggressively trimmed the log, invalidating in-flight offsets.
• Manual or external offset seek to a deleted/future offset. Tooling or a reset script set a committed offset outside the valid range.
• Reading from a topic that was recreated/truncated, so old committed offsets no longer correspond to existing data.
• Stored offset in an external store (custom offset management) that drifted past the log start.

How to Reproduce the Error

Create a short-retention topic, produce, let retention delete early segments, then seek to an old offset with reset disabled:

# topic with aggressive retention
kafka-topics.sh --create --topic short-lived --partitions 1 \
  --replication-factor 1 --config retention.ms=60000 --config segment.ms=10000 ...

# produce a few thousand records, wait > retention so early segments are deleted
# then start a consumer pinned to an old offset with reset disabled
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "none");
consumer.seek(new TopicPartition("short-lived", 0), 5); // offset 5 already deleted
consumer.poll(Duration.ofSeconds(1)); // throws OffsetOutOfRangeException

Diagnostic Commands

All commands are read-only. The key is comparing the committed offset against the partition’s current log start and end offsets.

# Committed offset vs LOG-END-OFFSET and lag for the group
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group payments --describe

# Earliest available offset per partition (the log start offset)
kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic payments --time -2

# Latest offset per partition (the log end offset)
kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic payments --time -1

# Topic retention settings that may have deleted the data
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name payments --describe

# Confirm the reset/out-of-range event in the consumer log
journalctl -u payment-worker --since "1 hour ago" | grep -iE "out of range|Resetting offset"

Step-by-Step Resolution

1. Locate the gap. Compare the group’s CURRENT-OFFSET from --describe with the earliest offset from GetOffsetShell --time -2. If the committed offset is below the earliest available offset, retention deleted the data and the position is genuinely invalid.

2. Pick the right reset policy for the consumer. With auto.offset.reset=none you opted into handling this yourself. For most consumers, set:

   • earliest — reprocess everything still retained (no data skipped, but possible reprocessing). Use when completeness matters and processing is idempotent.
   • latest — skip the gap and resume at the tail. Use when freshness matters more than the missed records.

   props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

3. Handle the exception explicitly if you keep none. Catch OffsetOutOfRangeException, read the affected partitions, and seek deliberately:

   try {
       records = consumer.poll(Duration.ofSeconds(1));
   } catch (OffsetOutOfRangeException e) {
       consumer.seekToBeginning(e.partitions()); // or seekToEnd
   }

4. Address the root cause: lag vs retention. If the consumer fell behind, scale it or speed up processing so it stays inside the retention window. If retention is simply too short for your worst-case downtime, raise retention.ms (or retention.bytes) on the topic.

5. Re-check after recovery. Re-run --describe and confirm CURRENT-OFFSET is now within [earliest, latest] and lag is shrinking.

Prevention and Best Practices

• Size topic retention to comfortably exceed your maximum realistic consumer downtime plus catch-up time.
• Alert on consumer lag approaching the point where it could outrun retention, not just on absolute lag.
• Choose auto.offset.reset deliberately per consumer: earliest for completeness, latest for freshness; reserve none for consumers that truly must fail loudly.
• Make processing idempotent so an earliest reset (and reprocessing) is safe.
• Treat retention config changes as risky: shortening retention.ms can instantly invalidate the offsets of any lagging group.
• Monitor kafka.log:type=Log,name=LogStartOffset so you can see when retention is advancing faster than consumers.

Related Errors

• NoOffsetForPartitionException — thrown with auto.offset.reset=none when there is no committed offset at all (new group), as opposed to one that is out of range.
• RecordTooLargeException — a different fetch-side error, about record size rather than offset position.
• CommitFailedException — unrelated commit-side rejection during rebalance.
• UnknownTopicOrPartitionException — the topic/partition no longer exists, a stronger failure than an out-of-range offset.

Frequently Asked Questions

Will earliest cause duplicate processing? Possibly. earliest resets to the oldest retained offset, which may be before some records you already processed if your committed offset was deleted. Idempotent processing makes this safe.

Why didn’t I get an exception, just a log line about resetting? Because auto.offset.reset was earliest or latest. The exception is only surfaced to your code when the policy is none. The reset is the broker/client automatically choosing a valid position.

How do I know retention deleted my data? Compare the committed offset to the earliest available offset (GetOffsetShell --time -2). If committed < earliest, the segment holding it was deleted by retention.

Can a too-new offset also trigger this? Yes, an offset above the log end offset is also out of range, though it is rarer — usually from manual seeks or external offset stores rather than normal consumption.