🚨 Automation Error: GitOps App Stuck in an OutOfSync Drift Loop

Overview

A GitOps application never settles into Synced. Argo CD (or Flux) reports a diff against Git, self-heal reapplies the manifest, something in the cluster mutates it right back, and the loop repeats every few seconds — forever:

$ argocd app get payments
Name:            payments
Sync Status:     OutOfSync from  (HEAD ab0123)
Health Status:   Progressing

GROUP  KIND        NAMESPACE  NAME      STATUS     HEALTH
apps   Deployment  prod       payments  OutOfSync  Progressing

$ argocd app diff payments
===== apps/Deployment prod/payments ======
<   replicas: 3
>   replicas: 7

Git says 3 replicas, the live object says 7, Argo syncs it back to 3, the HPA scales it to 7, and the application oscillates. The sync history shows dozens of syncs per minute and the controller is burning CPU going nowhere.

Symptoms

• An application shows OutOfSync continuously and never reaches a stable Synced state.
• Sync operations run in a tight loop; argocd app history (or the Flux Kustomization reconcile log) grows constantly.
• A live field flaps between the Git value and a controller-set value (replicas, resource requests, annotations, spec.clusterIP).
• Controller CPU and API-server request rate rise noticeably from the reconcile churn.
• kubectl get events shows repeated Scaled/Updated/Patched events on the same object.
• Rollouts appear to “flap” — pods scale up then get scaled down moments later.

Common Root Causes

• Two controllers own the same field. The classic case: Git pins spec.replicas while a HorizontalPodAutoscaler owns replicas. Each reverts the other, producing a permanent drift loop.
• Server-side defaulting / mutating admission. The API server or a mutating webhook injects defaults (a defaulted protocol: TCP, clusterIP, creationTimestamp, service-account token volumes) that aren’t in Git, so the live object always differs.
• Missing ignoreDifferences. Argo isn’t told to ignore controller-managed fields, so it treats every controller mutation as drift.
• Non-deterministic manifests. A Helm chart or Kustomize build renders slightly differently each run (random suffixes, timestamps, unsorted maps), so the desired state itself keeps changing.
• selfHeal fighting a legitimate external owner. Automated self-heal reverts changes made by another operator that genuinely owns the field.
• A webhook or operator that rewrites the resource (service mesh sidecar injection, a policy controller adding labels) after each sync.

Diagnostic Workflow

First, see exactly which field is drifting — the diff is the whole investigation:

argocd app diff payments
argocd app get payments -o json | jq '.status.sync.status, .status.conditions'

For Flux, inspect the Kustomization and its reconcile status:

flux get kustomization payments
kubectl -n flux-system describe kustomization payments | sed -n '/Events/,$p'

Confirm how often it is syncing — a loop shows many recent operations:

argocd app history payments | tail -20
kubectl -n argocd logs deploy/argocd-application-controller --since=5m \
  | grep -i 'payments' | grep -i 'sync\|compared\|diff'

Identify who else is writing the contested field using managed-field ownership:

kubectl -n prod get deploy payments --show-managed-fields -o yaml \
  | grep -A3 'manager:'
# Look for both argocd-controller AND another manager (e.g. kube-controller-manager for replicas)

Check for a controller that owns the field, e.g. an HPA on the same Deployment:

kubectl -n prod get hpa
kubectl -n prod describe hpa payments | grep -i 'reference\|current\|desired'

Watch the field flap in real time to confirm the loop:

kubectl -n prod get deploy payments -w -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas

Example Root Cause Analysis

A payments Deployment was managed by Argo CD with automated sync and selfHeal: true. The team also ran an HPA to handle traffic spikes. The Deployment manifest in Git still contained replicas: 3 from before the HPA existed.

kubectl get deploy payments --show-managed-fields revealed two managers writing spec.replicas: argocd-application-controller and kube-controller-manager (the HPA). Under load the HPA scaled to 7; Argo immediately saw OutOfSync (Git wanted 3), self-healed back to 3; the HPA scaled to 7 again. argocd app history showed a sync roughly every 10 seconds and the API server logged a stream of scale operations.

The root cause was dual ownership of spec.replicas. The correct GitOps pattern is: whoever autoscales owns the replica count, and Git must stop declaring it. Two changes fixed it.

First, remove replicas from the Git manifest so Argo no longer has an opinion about it, and tell Argo to ignore that field for good measure:

# argocd Application spec
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: payments
      namespace: prod
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Second, delete the replicas: line from the Deployment YAML in Git and commit, so the HPA is the sole owner of the field:

# in the app repo
sed -i '/^  replicas: 3$/d' apps/payments/deployment.yaml
git commit -am "payments: let HPA own replicas, remove from Git and ignore in Argo"

After the change, argocd app diff payments showed no difference on replicas, the app settled into Synced, and the sync-per-10-seconds churn dropped to the normal reconcile interval. The HPA scaled freely without Argo fighting it.

Prevention Best Practices

• One owner per field. Never declare a value in Git that a controller also manages (replicas under an HPA, clusterIP, VPA-managed resources). Pick one owner and remove the field from the other.
• Use ignoreDifferences for controller-managed and server-defaulted fields. Add jsonPointers/jqPathExpressions for HPA replicas, defaulted service fields, and injected sidecar/annotation noise.
• Make manifest rendering deterministic. Pin chart versions, avoid random suffixes and timestamps in generated manifests, and sort keys so the desired state doesn’t change between renders.
• Enable selfHeal deliberately. It’s powerful but will fight any legitimate external owner; scope it with ignoreDifferences so it only reverts true drift.
• Watch sync frequency as a signal. Alert when an app’s sync count per hour exceeds a threshold — a drift loop announces itself as sync churn long before anyone files a ticket.
• Reconcile with mutating webhooks in mind. If a policy or mesh controller rewrites resources post-sync, add those fields to ignoreDifferences or move the mutation into the source of truth.

Quick Command Reference

# See exactly what is drifting
argocd app diff my-app
flux get kustomization my-app

# Confirm a sync loop (many recent operations)
argocd app history my-app | tail -20

# Find dual ownership of a field
kubectl -n prod get deploy my-app --show-managed-fields -o yaml | grep -B1 -A3 'manager:'

# Common culprit: HPA vs Git-pinned replicas
kubectl -n prod get hpa

# Ignore a controller-managed field in Argo
# spec.ignoreDifferences -> jsonPointers: [ /spec/replicas ]

# Watch the field flap live
kubectl -n prod get deploy my-app -w -o custom-columns=NAME:.metadata.name,REPLICAS:.spec.replicas

Conclusion

A perpetual OutOfSync drift loop is almost never a bug in the GitOps engine — it’s two controllers arguing over one field, or the reconciler comparing against server-defaulted noise. The durable fix is to establish a single owner for every field: strip Git of values that autoscalers or the API server manage, and use ignoreDifferences to silence the rest. Then treat sync frequency as a first-class signal, because a healthy application reconciles on a calm cadence, and one syncing every ten seconds is quietly telling you it will never be done.