📦 Thanos thanos_shipper_uploads_total Not Increasing: Blocks Not Shipping to Object Storage

What the Metric Means

thanos_shipper_uploads_total is a counter incremented by the Thanos shipper each time it successfully uploads a TSDB block to object storage. It lives in the Thanos sidecar (next to Prometheus) and in the Thanos receiver. On a healthy system it steps up roughly every two hours, because Prometheus cuts a new 2h block on that cadence and the shipper ships it to your S3, GCS, or Azure bucket.

A flat thanos_shipper_uploads_total is different from a rising thanos_shipper_upload_failures_total. Failures mean uploads are being attempted and rejected. A flat uploads counter with no corresponding failures usually means the shipper is not even trying — there is nothing new to ship, shipping is misconfigured, or the shipper is not running at all. This guide focuses on that “nothing is happening” case.

Exact Symptom

increase(thanos_shipper_uploads_total[6h]) == 0

Over a multi-hour window you expect at least two or three increments. If the value is dead flat while failures stay at zero:

increase(thanos_shipper_upload_failures_total[6h]) == 0

then blocks simply are not being handed to the shipper — the problem is upstream of the object-storage call.

Common Causes

• Not enough time has passed — Prometheus only ships a block after it is cut and compacted, so a freshly started sidecar can legitimately take up to ~2h before the first upload.
• Wrong block-duration flags — if --storage.tsdb.min-block-duration does not equal --storage.tsdb.max-block-duration, Prometheus performs local compaction and Thanos deliberately refuses to upload compacted blocks, so uploads never start.
• Shipping disabled — the sidecar was started without --objstore.config/--objstore.config-file, so the shipper is inert by design.
• No new samples — if Prometheus is not scraping anything, no head block fills, so no 2h block is ever cut.
• --shipper.upload-compacted expectations — for the receiver or backfill scenarios, compacted blocks only ship when this flag is set.
• TSDB retention too short — an aggressively low --storage.tsdb.retention.time can delete blocks before the shipper syncs them.
• Shipper meta file stuck — a stale or unwritable thanos.shipper.json can leave the shipper believing everything is already uploaded.

Diagnostic Commands

Confirm whether any blocks exist locally yet and how old the head is:

ls -la /prometheus/  # look for ULID-named block directories, not just wal/ and chunks_head/

Check that the sidecar was actually given an object-storage config:

kubectl get pod prometheus-0 -o yaml | grep -E "objstore.config|shipper"

Verify the block-duration flags on Prometheus, the single most common cause:

kubectl get pod prometheus-0 -o jsonpath='{.spec.containers[?(@.name=="prometheus")].args}' \
  | tr ',' '\n' | grep block-duration

Read the sidecar log for shipper activity (or the lack of it):

kubectl logs prometheus-0 -c thanos-sidecar --tail=200 | grep -i shipper

List what is already in the bucket to compare against local blocks:

thanos tools bucket ls --objstore.config-file=/etc/thanos/objstore.yml

Step-by-Step Resolution

1. Rule out “too early.” If the sidecar started less than ~2–3 hours ago and there are no ULID block directories under the TSDB path yet, wait for Prometheus to cut its first block. This is normal.

2. Confirm shipping is enabled. The sidecar must be launched with an object-storage config; without it the shipper does nothing:

args:
  - "sidecar"
  - "--tsdb.path=/prometheus"
  - "--prometheus.url=http://localhost:9090"
  - "--objstore.config-file=/etc/thanos/objstore.yml"

3. Fix block durations. Set min and max block duration equal to 2h so Prometheus does not compact locally and Thanos can ship raw blocks:

# prometheus container args
args:
  - "--storage.tsdb.min-block-duration=2h"
  - "--storage.tsdb.max-block-duration=2h"

If these differ, Prometheus compacts blocks and the shipper skips them, so thanos_shipper_uploads_total never moves.

4. Verify Prometheus is scraping. No targets means no samples, means no block. Check active targets:

count(up == 1)

5. Check retention vs. shipping cadence. Ensure --storage.tsdb.retention.time is comfortably larger than the block/upload interval so blocks are not deleted before the shipper uploads them.

6. Validate the bucket path end to end to confirm the sidecar can reach and read storage:

thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore.yml

7. Confirm recovery. After the fix, watch the counter step up on the next block boundary:

increase(thanos_shipper_uploads_total[3h]) > 0

Prevention and Best Practices

• Alert on a flat uploads counter — increase(thanos_shipper_uploads_total[6h]) == 0 catches silent stalls that a failures-only alert misses entirely.
• Standardize the sidecar+Prometheus flag set (2h min = 2h max, objstore config present) in your Helm values or manifests so no environment drifts.
• Keep TSDB retention generous relative to your shipping interval so blocks survive long enough to upload.
• Watch both thanos_shipper_uploads_total (progress) and thanos_shipper_upload_failures_total (errors) together; the pair distinguishes “not trying” from “trying and failing.”
• Use thanos tools bucket ls in CI or a health check to confirm blocks continue to arrive in object storage.

The DevOps AI prompt library has prompts that turn a flat shipper counter and a pod spec into a prioritized checklist.

Frequently Asked Questions

Why is thanos_shipper_uploads_total flat but there are no failures? The shipper is not being handed any blocks to upload. That usually means shipping is disabled (no objstore config), no new 2h block has been cut yet, or local compaction is producing blocks Thanos deliberately skips.

How long until the first upload after starting the sidecar? Prometheus cuts a block roughly every two hours, and the shipper uploads shortly after. A freshly started sidecar can legitimately show a flat counter for up to two to three hours.

Does the sidecar upload without an objstore config? No. If the sidecar is started without --objstore.config or --objstore.config-file, the shipper is inert and thanos_shipper_uploads_total stays at zero by design.

Why does setting min and max block duration equal fix this? When they differ, Prometheus performs local compaction, and Thanos refuses to upload compacted blocks. Setting both to 2h keeps blocks raw so the shipper can ship them.

How do I confirm blocks are actually reaching the bucket? Run thanos tools bucket ls against the same objstore config and compare the listed ULIDs to the block directories on the local TSDB path.

Prometheus & monitoring guides