🚨 Argo CD Sync Fails: Hook Job BackoffLimitExceeded Blocks GitOps Deploy

Overview

Argo CD runs resource hooks — PreSync, Sync, and PostSync Jobs — as part of a sync operation. When a hook Job’s pods fail repeatedly and the Job exhausts its backoffLimit, the Job reports BackoffLimitExceeded, the sync is marked failed or stuck in Progressing, and the whole GitOps rollout halts:

Job.batch "db-migrate-presync" is in failed state: BackoffLimitExceeded
one or more synchronization tasks are not valid
Operation has failed: hook job db-migrate-presync failed

In the Argo CD UI the Application shows OutOfSync/Failed with the hook resource red, and the sync will not advance to the application manifests until the hook succeeds or is removed.

Symptoms

• Argo CD sync stops at a PreSync/PostSync hook and never reaches the app resources.
• A hook Job shows BackoffLimitExceeded in kubectl get job and Failed conditions.
• The hook’s pods are in Error or CrashLoopBackOff, or were OOMKilled.
• The Application status is Failed with message hook job <name> failed.
• Retried syncs fail identically because the underlying Job command still errors.

Common Root Causes

• The hook command genuinely fails — a database migration, schema check, or smoke test inside the hook exits non-zero on every attempt.
• Missing dependency at hook time — the hook runs before the resource it needs exists (e.g. a migration Job runs before the DB Service/secret is created).
• Bad image or config — wrong image tag, missing env var, or an unmounted secret makes the pod fail on startup.
• backoffLimit too low — a transient failure (DB not yet reachable) exhausts a backoffLimit: 0 before the dependency is ready.
• OOM or resource limits — the hook pod is OOMKilled and every retry dies the same way.
• Leftover hook resource — a previous failed Job with the same name blocks recreation because the hook delete-policy did not clean it up.

Diagnostic Workflow

Identify the Application and the failing hook resource:

argocd app get <app> --show-operation
kubectl -n <ns> get jobs -l argocd.argoproj.io/instance=<app>

Inspect the failed Job and its conditions:

kubectl -n <ns> get job <hook-job> -o jsonpath='{.status.conditions}' | jq
kubectl -n <ns> describe job <hook-job>

Read the actual failure from the pod logs — this is where the real error lives:

kubectl -n <ns> get pods --selector job-name=<hook-job>
kubectl -n <ns> logs job/<hook-job> --all-containers --tail=200
kubectl -n <ns> describe pod <hook-pod> | grep -iA3 'State\|Reason\|Exit'

Check the hook annotations to understand phase and cleanup policy:

kubectl -n <ns> get job <hook-job> -o jsonpath='{.metadata.annotations}' | jq
# argocd.argoproj.io/hook: PreSync
# argocd.argoproj.io/hook-delete-policy: HookSucceeded

Review the Argo CD controller view of the operation:

argocd app get <app> -o json | jq '.status.operationState.message'

Example Root Cause Analysis

A team’s PreSync hook db-migrate-presync failed every sync with BackoffLimitExceeded. kubectl get job showed 0 successful completions and backoffLimit: 2. Pod logs revealed the truth:

error: could not connect to database: dial tcp 10.0.14.3:5432: connect: connection refused

The migration hook was scheduled in the same sync wave as the Postgres StatefulSet, so it ran before the database was accepting connections. With backoffLimit: 2, all three attempts fired within ~40 seconds — long before Postgres was ready — and the Job gave up.

The fix had two parts: move the migration hook to a later sync wave with argocd.argoproj.io/sync-wave: "1" so the database provisions first, and raise backoffLimit and add an init wait so a slow-starting DB does not exhaust retries. After correcting the ordering, the hook connected on its first attempt and the sync completed. The leftover failed Job was cleared with kubectl delete job db-migrate-presync before the retry so the new hook could be created.

Prevention Best Practices

• Order hooks with argocd.argoproj.io/sync-wave so migrations and checks run after their dependencies are provisioned.
• Make hook commands wait for their dependencies (an init container or a wait-for loop) rather than relying on retry timing.
• Set a realistic backoffLimit and activeDeadlineSeconds so transient slowness does not exhaust retries but a truly broken hook still fails fast.
• Set hook-delete-policy: HookSucceeded (or BeforeHookCreation) so leftover failed Jobs do not block recreation.
• Give hook pods resource requests/limits sized to avoid OOMKilled, and log enough to diagnose from kubectl logs alone.
• Test hooks in a staging Application before they gate production syncs.

Quick Command Reference

# See the failing operation and hooks
argocd app get <app> --show-operation
kubectl -n <ns> get jobs -l argocd.argoproj.io/instance=<app>

# Root-cause the Job
kubectl -n <ns> describe job <hook-job>
kubectl -n <ns> logs job/<hook-job> --all-containers --tail=200

# Inspect hook annotations (phase, delete policy, wave)
kubectl -n <ns> get job <hook-job> -o jsonpath='{.metadata.annotations}' | jq

# Clear a leftover failed hook Job, then retry
kubectl -n <ns> delete job <hook-job>
argocd app sync <app>

Conclusion

BackoffLimitExceeded on an Argo CD hook Job almost never means Argo CD is broken — it means the hook’s own command failed every attempt, and the pod logs will tell you why. The common trap is retry timing: a hook that runs before its dependency is ready burns through a low backoffLimit and stalls the entire sync. Fix the ordering with sync-waves, make hooks wait for what they need, set sane retry limits, and ensure failed Jobs are cleaned up so the next sync starts clean.