📦 Thanos thanos_shipper_upload_failures_total Rising: Fix Failed Block Uploads

What the Metric Means

thanos_shipper_upload_failures_total is a counter exposed by the Thanos shipper — the component embedded in the Thanos sidecar (alongside Prometheus) and in the Thanos receiver. The shipper watches the local TSDB directory for completed 2h blocks and uploads them to object storage (S3, GCS, Azure Blob, and compatible backends). Every time an upload attempt fails, this counter increments.

A healthy shipper keeps this counter flat while thanos_shipper_uploads_total climbs. When thanos_shipper_upload_failures_total starts rising, blocks are being produced locally but are not landing in your bucket. Because Thanos Store Gateway and Compactor only see what is in object storage, sustained upload failures mean gaps in long-term history and, eventually, a full local disk on the Prometheus node.

Exact Symptom

rate(thanos_shipper_upload_failures_total[5m]) > 0

In the sidecar logs you will typically see something like:

level=error ts=2026-07-15T09:41:22.109Z caller=shipper.go:391
  msg="upload failed" block=01J8...ABC
  err="upload object: Access Denied: s3: AccessDenied status code: 403"

The err= string is the single most important clue — it names the underlying object-storage operation that failed.

Common Causes

• Bad or missing credentials — expired IAM keys, wrong access key/secret, or an IRSA/workload-identity role that lacks s3:PutObject.
• Insufficient bucket permissions — the identity can list but not write, or is missing s3:PutObject/storage.objects.create.
• Wrong endpoint or region — a MinIO/Ceph endpoint typo, or an S3 region mismatch that yields 301 PermanentRedirect or AuthorizationHeaderMalformed.
• Clock skew — S3 signature validation rejects requests when the node clock drifts more than ~15 minutes (RequestTimeTooSkewed).
• Read-only or full filesystem — the shipper cannot stage the block or write its thanos.shipper.json meta file.
• Network egress blocked — a security group, egress NetworkPolicy, or missing NAT/VPC endpoint drops connections to the storage API.
• Block-duration misconfiguration — for the sidecar, --storage.tsdb.min-block-duration must equal --storage.tsdb.max-block-duration (both 2h) or Prometheus applies local compaction that Thanos refuses to upload.

Diagnostic Commands

First, confirm the shipper is actually failing rather than idle:

increase(thanos_shipper_upload_failures_total[15m])

Then correlate with the lower-level bucket operation counter, which breaks failures down by operation (upload, iter, get):

sum by (operation) (rate(thanos_objstore_bucket_operation_failures_total[5m]))

Read the sidecar logs for the concrete error:

kubectl logs -l app.kubernetes.io/name=prometheus -c thanos-sidecar --tail=200 \
  | grep -Ei "upload|objstore|denied|skew"

Validate the object-storage config independently of Prometheus using the thanos tools bucket verify subcommand, which round-trips a read/write against the same objstore.config:

thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore.yml

Step-by-Step Resolution

1. Read the err= string. It tells you whether the problem is auth (AccessDenied, 403), routing (301, PermanentRedirect), time (RequestTimeTooSkewed), or connectivity (dial tcp ... i/o timeout). Everything below is chosen based on this.

2. Verify the objstore config. Confirm bucket name, endpoint, and region are correct:

type: S3
config:
  bucket: "thanos-blocks-prod"
  endpoint: "s3.us-east-1.amazonaws.com"
  region: "us-east-1"
  aws_sdk_auth: true

3. Test permissions directly with the same credentials the sidecar uses, so you separate a policy problem from a Thanos problem:

aws s3 cp /tmp/probe.txt s3://thanos-blocks-prod/probe.txt

If this fails with AccessDenied, fix the IAM policy (grant s3:PutObject, s3:GetObject, s3:ListBucket on the bucket and its objects) before touching Thanos.

4. Fix clock skew if the error is RequestTimeTooSkewed. Ensure NTP/chrony is running on the node:

timedatectl status | grep "System clock synchronized"

5. Correct the block-duration flags on the sidecar’s Prometheus if uploads never succeed and logs mention non-2h blocks:

args:
  - "--storage.tsdb.min-block-duration=2h"
  - "--storage.tsdb.max-block-duration=2h"

Setting them equal disables local compaction so Thanos can safely ship raw 2h blocks.

6. Confirm recovery. Once fixed, the shipper retries failed blocks automatically. Watch the failure rate flatten and uploads resume:

rate(thanos_shipper_upload_failures_total[5m])

7. Re-run thanos tools bucket verify to confirm the backend accepts writes cleanly.

Prevention and Best Practices

• Alert on rate(thanos_shipper_upload_failures_total[5m]) > 0 sustained for 15m, and separately on thanos_shipper_uploads_total failing to increase, so you catch both hard failures and silent stalls.
• Prefer IRSA / workload identity / managed identities over long-lived static keys so credentials do not expire under you.
• Keep NTP running everywhere; clock skew is a recurring, easily missed cause of S3 signature failures.
• Monitor local disk on the Prometheus node — sustained upload failures cause blocks to accumulate locally.
• Always set the sidecar’s min and max block duration to 2h; never let Prometheus compact locally when Thanos is shipping.

Speed up triage with the DevOps AI prompt library, which includes ready-made prompts for decoding Thanos and object-storage errors.

Frequently Asked Questions

What does thanos_shipper_upload_failures_total measure? It counts failed attempts by the Thanos shipper to upload a completed 2h TSDB block to object storage. A rising counter means blocks are not reaching your S3/GCS/Azure bucket.

Will Thanos retry a failed upload? Yes. The shipper reattempts unshipped blocks on its next sync interval, so once you fix the root cause (credentials, endpoint, clock) the backlog uploads automatically without manual intervention.

Why do I see AccessDenied even though the bucket exists? The identity can usually list the bucket but lacks write permission. Grant s3:PutObject (or storage.objects.create on GCS) on the bucket’s objects, and verify with a direct aws s3 cp using the same credentials.

Does clock skew really break uploads? Yes. S3-compatible backends reject signed requests when the node clock drifts more than roughly 15 minutes, returning RequestTimeTooSkewed. Run NTP/chrony to keep time in sync.

How is this different from thanos_objstore_bucket_operation_failures_total? The objstore counter is lower level and breaks failures down by operation (upload, get, iter). Use it to confirm whether the failing operation is the upload itself or something else in the storage path.

Prometheus & monitoring guides