🚨 Kubernetes Error Guide: 'upstream connect error or disconnect/reset before headers' Envoy

Exact Error Message

When an Envoy-based proxy (Istio ingress gateway, sidecar, or a standalone Envoy) cannot establish a TCP/TLS connection to its upstream, it returns a 503 with a distinctive body:

HTTP/1.1 503 Service Unavailable
content-length: 95
content-type: text/plain
date: Sun, 28 Jun 2026 15:11:44 GMT
server: istio-envoy

upstream connect error or disconnect/reset before headers. reset reason: connection failure

The matching Envoy access-log entry exposes the response flags that pinpoint the cause:

[2026-06-28T15:11:44.812Z] "GET /api HTTP/1.1" 503 UF,URX "-" 0 95 1 - "10.0.4.9" "curl/8.5.0" "a1b2..." "api.svc:8080" "10.0.2.31:8080" outbound|8080||api.default.svc.cluster.local - 10.0.2.31:8080 10.0.4.9:51324 - default

UF (upstream connection failure) and URX (upstream reset before response) are the flags that tell you Envoy could not complete the connection.

What the Error Means

Envoy accepted the downstream request, selected an upstream endpoint, and then failed during connection setup — before a single response header came back. reset reason: connection failure means the TCP connect or the TLS handshake to the chosen pod IP did not succeed. This is a Layer 4 / TLS problem between the proxy and the backend, not an application-level error: your app code never ran.

In a mesh this most often comes down to a mutual-TLS expectation mismatch — Envoy dialing the backend with a client certificate that the backend is not configured to accept, or vice versa.

Common Causes

• mTLS mode mismatch — a PeerAuthentication of STRICT on the server while the client side sends plaintext, or a DestinationRule with tls.mode: DISABLE/SIMPLE while the server expects ISTIO_MUTUAL.
• Backend application is down — the pod is in CrashLoopBackOff or listening on a different port than Envoy dials.
• Wrong port / appProtocol — the Service port name or appProtocol causes Envoy to speak HTTP to a TCP-only port, or dial a port nothing listens on.
• No healthy endpoints — the upstream cluster has zero endpoints (Service selector mismatch or pods not Ready), so Envoy fails fast.
• Sidecar not injected — the backend has no Envoy sidecar but the mesh expects one, so mTLS to it cannot complete.
• NetworkPolicy or firewall drops the proxy-to-pod connection.

How to Reproduce the Error

Set STRICT mTLS on a workload, then call it from a pod without a sidecar:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: api-strict
  namespace: default
spec:
  selector:
    matchLabels: { app: api }
  mtls:
    mode: STRICT

kubectl apply -f peerauth.yaml
# Call from a plain pod (no Envoy sidecar) -> plaintext rejected
kubectl run probe --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s http://api.default.svc.cluster.local:8080/
# -> upstream connect error or disconnect/reset before headers. reset reason: connection failure

The server rejects the non-mTLS connection, Envoy reports a connection failure, and the caller gets the 503 body above.

Diagnostic Commands

# Read the response flags (UF/URX) from the proxy access log
kubectl logs <CLIENT_POD> -c istio-proxy --tail=30

# Does the upstream cluster have endpoints, and what TLS does it expect?
istioctl proxy-config endpoints <CLIENT_POD> --cluster 'outbound|8080||api.default.svc.cluster.local'
istioctl proxy-config cluster <CLIENT_POD> --fqdn api.default.svc.cluster.local -o json | grep -i tls

# Check effective mTLS mode on the destination
istioctl x describe pod <SERVER_POD>

# Confirm the backend is actually listening and Ready
kubectl get pods -l app=api -o wide
kubectl get endpoints api

# Is a sidecar injected on the server?
kubectl get pod <SERVER_POD> -o jsonpath='{.spec.containers[*].name}'; echo

The access-log response flags plus istioctl x describe together tell you whether this is an mTLS mismatch, a dead backend, or an empty cluster.

Step-by-Step Resolution

1. Read the response flags. UF/URX = connection-level failure (this guide). NR = no route. UH = no healthy upstream. Knowing the flag routes you to the right fix immediately.

2. Rule out a dead backend first. Confirm the pod is Ready and listening on the dialed port:

kubectl get endpoints api -o wide
kubectl logs <SERVER_POD> -c <APP_CONTAINER> --tail=20

If endpoints are empty or the app is crashing, fix that before touching mesh config.

3. Reconcile mTLS expectations. With PeerAuthentication: STRICT, every client must use a sidecar and ISTIO_MUTUAL. Either inject a sidecar on the caller, set the DestinationRule tls.mode to ISTIO_MUTUAL, or relax the server to PERMISSIVE while you migrate:

istioctl proxy-config cluster <CLIENT_POD> --fqdn api.default.svc.cluster.local -o json | grep -A2 transportSocket

A DestinationRule saying DISABLE against a STRICT server is the classic contradiction — align them.

4. Verify port and appProtocol. Ensure the Service targetPort matches the container port and that a TCP service is not labelled as HTTP. Mis-declared appProtocol makes Envoy speak the wrong protocol and the connection resets.

5. Confirm sidecar injection. The server pod must contain an istio-proxy container if the mesh enforces mTLS to it. If missing, enable injection on the namespace and restart the workload.

6. Re-test. Re-issue the request; a clean 200 (or your app’s real status) confirms the connection now establishes.

Prevention and Best Practices

• Roll mTLS out in PERMISSIVE mode first, confirm all callers negotiate mutual TLS, then flip to STRICT — never the reverse.
• Keep PeerAuthentication and DestinationRule TLS modes managed together; a server going STRICT without matching client DestinationRules guarantees connection failures.
• Alert on Envoy UF/UH/URX response flags in your access logs; a spike is an early warning of a backend or mesh-config outage.
• Standardise sidecar injection at the namespace level so no workload silently joins the mesh without a proxy.
• Name Service ports correctly (http, grpc, tcp) so Envoy applies the right protocol. More patterns in our Kubernetes & Helm guides.

Related Errors

• 503 Service Temporarily Unavailable from ingress-nginx — the nginx-side equivalent of an empty upstream.
• No endpoints available for service — the UH no-healthy-upstream variant.
• x509 certificate signed by unknown authority — a related TLS trust failure.

Frequently Asked Questions

What is the difference between connection failure and connection termination? connection failure means the TCP/TLS connect never completed (backend down, wrong port, mTLS rejected). connection termination means a connection was established and then the upstream cut it mid-stream, which points more at the app closing the socket or a timeout.

It worked until I enabled STRICT mTLS — why? STRICT forbids plaintext. Any caller without a sidecar, or any DestinationRule with tls.mode: DISABLE/SIMPLE, now fails the handshake and Envoy reports a connection failure. Move those callers to ISTIO_MUTUAL or revert to PERMISSIVE.

The backend is healthy and mTLS is consistent, but I still get the reset. Check the port and appProtocol. If Envoy treats a raw TCP/gRPC port as HTTP/1.1 (or vice versa), the handshake resets before headers. Confirm with istioctl proxy-config listener and correct the Service port name.

Does a 503 with this body ever come from my application? No. The server: istio-envoy header and the exact reset reason text are emitted by Envoy itself. If your app produced the 503, the body and server header would be your application’s, and the proxy access log would show a real HTTP status from the upstream.