📦 Debugging Thanos Sidecar Uploads to Object Storage (S3/GCS/Azure)

How Thanos Sidecar Uploads Actually Work

Before debugging, it helps to know the exact pipeline. Prometheus runs with the Thanos sidecar beside it. Every ~2 hours Prometheus cuts a completed TSDB block into its --storage.tsdb.path directory as a ULID-named folder. The sidecar’s embedded shipper watches that directory, and for each new block it uploads the block’s chunks, index, and meta.json to object storage — S3, GCS, Azure Blob, or any compatible backend (MinIO, Ceph RGW). It records progress in a local thanos.shipper.json meta file so it never re-uploads the same block.

The backend is described by a single objstore.config (inline via --objstore.config or a file via --objstore.config-file). Downstream, the Thanos Store Gateway and Compactor read only what is in the bucket, so if uploads break, long-term queries and compaction silently lose data even though Prometheus itself looks healthy. This playbook walks the pipeline from metric to backend.

Start With the Two Shipper Metrics

These two counters tell you which failure mode you are in:

# Are uploads succeeding at all?
increase(thanos_shipper_uploads_total[6h])

# Are uploads being attempted and rejected?
rate(thanos_shipper_upload_failures_total[5m])

• Uploads increasing, failures zero → healthy, stop here.
• Uploads flat, failures rising → uploads are attempted but the backend rejects them (auth, routing, clock). Jump to the objstore section.
• Uploads flat, failures zero → nothing is being shipped (shipping disabled, no new blocks, wrong block duration). Check the pipeline upstream of storage.

Also inspect the lower-level bucket counter, which attributes failures to a specific operation:

sum by (operation) (rate(thanos_objstore_bucket_operation_failures_total[5m]))

Common Causes Across S3, GCS, and Azure

• Credentials — expired/incorrect keys, or an IRSA/workload-identity/managed-identity binding that lacks write permission.
• Permissions — missing s3:PutObject (AWS), storage.objects.create (GCS), or the Storage Blob Data Contributor role (Azure).
• Endpoint/region — a typo’d MinIO endpoint, an S3 region mismatch (301 PermanentRedirect), or a wrong Azure storage account name.
• Clock skew — S3 signature validation rejects requests when the node clock drifts (RequestTimeTooSkewed).
• Read-only / full filesystem — the shipper cannot stage blocks or write thanos.shipper.json.
• Network egress — security groups, egress NetworkPolicies, missing VPC/S3 endpoints, or proxy rules dropping storage-API traffic.
• Block-duration mismatch — --storage.tsdb.min-block-duration must equal --storage.tsdb.max-block-duration (2h) or local compaction blocks uploads.

Diagnostic Commands

Confirm the sidecar has an objstore config and the expected flags:

kubectl get pod prometheus-0 -o yaml | grep -E "objstore|block-duration|tsdb.path"

Pull the sidecar’s own view of the error:

kubectl logs prometheus-0 -c thanos-sidecar --tail=300 | grep -Ei "shipper|objstore|denied|skew|timeout"

Round-trip the backend independently with the same config the sidecar uses:

thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore.yml

List what has actually landed in the bucket:

thanos tools bucket ls -o table --objstore.config-file=/etc/thanos/objstore.yml

Step-by-Step Resolution

1. Classify the failure using the two shipper metrics above. Everything else branches from whether failures are rising or the counter is simply flat.

2. Read the logs for the concrete err=. The string names the failing operation and reason — auth, routing, time, or connectivity. Do not guess; let the error direct you.

3. Validate the objstore config for your backend. Examples:

# S3
type: S3
config:
  bucket: "thanos-blocks"
  endpoint: "s3.eu-west-1.amazonaws.com"
  region: "eu-west-1"

# GCS
type: GCS
config:
  bucket: "thanos-blocks"

# Azure
type: AZURE
config:
  storage_account: "thanosprod"
  container: "thanos-blocks"

4. Test permissions directly with the same identity the pod uses, isolating a policy problem from a Thanos problem:

aws s3 cp /tmp/probe.txt s3://thanos-blocks/probe.txt   # or: gsutil cp / az storage blob upload

5. Check the clock if the error is RequestTimeTooSkewed:

timedatectl status | grep "System clock synchronized"

6. Fix the block-duration flags on Prometheus so uploads can start:

args:
  - "--storage.tsdb.min-block-duration=2h"
  - "--storage.tsdb.max-block-duration=2h"

7. Check the local filesystem the sidecar writes to — a read-only or full volume stops staging and the meta file:

kubectl exec prometheus-0 -c thanos-sidecar -- df -h /prometheus
kubectl exec prometheus-0 -c thanos-sidecar -- touch /prometheus/.probe

8. Confirm recovery. The shipper retries unshipped blocks automatically. Watch failures flatten and uploads resume, then verify the bucket:

rate(thanos_shipper_upload_failures_total[5m])
increase(thanos_shipper_uploads_total[3h]) > 0

thanos tools bucket verify --objstore.config-file=/etc/thanos/objstore.yml

Prevention and Best Practices

• Instrument both shipper counters and thanos_objstore_bucket_operation_failures_total; together they tell you whether uploads are failing, stalled, or healthy.
• Alert on rate(thanos_shipper_upload_failures_total[5m]) > 0 for 15m and on a flat thanos_shipper_uploads_total over 6h.
• Prefer IRSA / workload identity / managed identities over static keys, and scope them to the minimum write permissions on one bucket.
• Keep NTP running on every node; clock skew is a common, easily missed S3 failure.
• Always set the sidecar’s Prometheus min and max block duration to 2h, and never enable local Prometheus compaction when Thanos ships blocks.
• Run thanos tools bucket verify after any credential, endpoint, or IAM change so you catch regressions before blocks pile up locally.

For faster root-cause on messy object-storage errors, the DevOps AI prompt library turns a sidecar log line into a prioritized fix list.

Frequently Asked Questions

Which comes first when debugging Thanos uploads? Start with the two shipper metrics. If thanos_shipper_upload_failures_total is rising, the backend is rejecting attempts; if thanos_shipper_uploads_total is flat with no failures, nothing is being shipped. That split decides everything downstream.

What does thanos tools bucket verify do? It round-trips read and write operations against the same objstore config the sidecar uses, so you can confirm credentials, endpoint, region, and permissions independently of Prometheus.

Why do downstream queries lose data when uploads fail? The Store Gateway and Compactor read only from object storage. If the sidecar cannot upload blocks, long-term history and compaction never see them, even though the local Prometheus keeps working normally.

Is the debugging different for S3, GCS, and Azure? The pipeline and shipper metrics are identical; only the objstore config block and the permission model differ (s3:PutObject on AWS, storage.objects.create on GCS, Storage Blob Data Contributor on Azure).

Why must min and max block duration be equal? If they differ, Prometheus compacts blocks locally, and Thanos refuses to upload compacted blocks. Setting both to 2h keeps blocks raw so the shipper can upload them.

Prometheus & monitoring guides