🚨 Grafana Error Guide: 'net/http: TLS handshake timeout' — Fix Datasource TLS Failures

Overview

Grafana raises this when a datasource query opens a TLS connection to a backend (Prometheus, Loki, a SQL database, an HTTPS proxy) but the TLS handshake does not complete within the client timeout. The panel shows an error and the server log records:

Post "https://prometheus.internal:9090/api/v1/query_range": net/http: TLS handshake timeout

Unlike a certificate error (x509: certificate signed by unknown authority), this is a timing failure: the TCP connection opened, but the TLS negotiation stalled — no ServerHello, or the exchange was too slow — so Go’s HTTP client aborted. The query never reached the backend.

Symptoms

• Panels intermittently show net/http: TLS handshake timeout; retries sometimes succeed.
• The datasource Save & test button fails with the same message.
• Only HTTPS datasources are affected; plain-HTTP ones work.
• The backend’s own logs show no incoming request for the failed query — it never arrived.
• Failures correlate with load, large responses, or a specific network path (VPN, service mesh, egress proxy).

Common Root Causes

• Slow or overloaded backend taking longer than the handshake timeout to respond to ClientHello (CPU-starved Prometheus/Loki during heavy queries).
• MTU / MSS mismatch on a tunnel (VPN, WireGuard, overlay network) fragmenting the TLS ClientHello/ServerHello so the handshake stalls.
• A TLS-terminating proxy or service mesh sidecar (Envoy, mTLS) that is slow, misconfigured, or intercepting traffic Grafana didn’t expect.
• Firewall/security group allowing the TCP SYN but silently dropping later packets, so the handshake hangs until timeout.
• dialTimeout/tlsHandshakeTimeout too low in Grafana’s datasource jsonData for a genuinely slow but working path.
• Wrong port — pointing HTTPS at a plain-HTTP port makes the server never send a ServerHello.

Diagnostic Workflow

1. Confirm it’s TLS timing, not certs. Read the Grafana server log around the failure:

journalctl -u grafana-server --since '15 min ago' | grep -i 'handshake timeout\|tls'
# or in-container
docker logs grafana 2>&1 | grep -i 'handshake timeout'

If the message is TLS handshake timeout (not x509), it’s timing/network, not trust.

2. Time the handshake from the Grafana host. This isolates Grafana from the network path:

# From the Grafana host/container, measure the handshake
curl -kv --max-time 10 https://prometheus.internal:9090/-/ready 2>&1 | grep -i 'connected\|ssl\|handshake'

# Precise handshake timing
curl -k -o /dev/null -s -w 'connect=%{time_connect} tls=%{time_appconnect}\n' \
  https://prometheus.internal:9090/-/ready

If time_appconnect is large or the command hangs, the problem is the path/backend, not Grafana.

3. Check for packet drops and MTU. A stalled handshake with a healthy TCP connect often means fragmentation:

# Does a large-packet, no-fragment ping fail while small ones succeed?
ping -M do -s 1472 prometheus.internal
# Test a smaller MSS/MTU path
ping -M do -s 1400 prometheus.internal

If 1472 fails but 1400 succeeds, lower the interface MTU (or clamp MSS) on the tunnel.

4. Verify the port actually speaks TLS.

openssl s_client -connect prometheus.internal:9090 -servername prometheus.internal </dev/null
# Immediate close with no ServerHello => wrong port / not HTTPS

5. Tune Grafana’s datasource timeouts in jsonData if the path is slow but valid. Provisioned example:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: https://prometheus.internal:9090
    jsonData:
      timeout: 60          # overall query timeout (seconds)
      dialTimeout: 10      # TCP dial
      tlsHandshakeTimeout: 15
      keepAlive: 30
    secureJsonData: {}

Restart Grafana (or reload provisioning) and re-test.

6. Test through the API to reproduce without the browser:

curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/datasources/uid/$UID/health | jq

Example Root Cause Analysis

A team ran Grafana in Kubernetes querying Prometheus across a WireGuard tunnel to another cluster. Panels intermittently failed with net/http: TLS handshake timeout, worst on large query_range calls. Certs were valid and small dashboards worked.

curl from the Grafana pod connected instantly (time_connect low) but time_appconnect sometimes hung for the full 10s. ping -M do -s 1472 across the tunnel failed while -s 1400 succeeded — the WireGuard interface MTU (1420) meant the 1500-byte ServerHello fragments were dropped, stalling only the larger handshakes.

Fix: clamp MSS on the tunnel (iptables ... TCPMSS --clamp-mss-to-pmtu) and set the pod’s route MTU to 1420. Handshakes completed immediately afterward and the timeouts disappeared — no Grafana or Prometheus change was needed.

Prevention Best Practices

• Match MTU/MSS across tunnels and overlays so TLS handshake packets never fragment; clamp MSS to path MTU on VPN/mesh links.
• Set realistic timeout/tlsHandshakeTimeout in datasource jsonData for genuinely slow backends instead of leaving defaults that abort valid handshakes.
• Monitor backend latency (Prometheus/Loki query duration) and alert before the backend gets slow enough to miss the handshake window.
• Keep TLS-terminating proxies and mesh sidecars healthy, and confirm they’re actually in the path you think they are.
• Verify firewall rules allow the full flow, not just the SYN — silent mid-handshake drops look exactly like this error.
• Point datasources at the correct HTTPS port and confirm with openssl s_client during setup.

Quick Command Reference

# Read the error in Grafana logs
journalctl -u grafana-server --since '15 min ago' | grep -i 'handshake timeout'

# Time the TLS handshake from the Grafana host
curl -k -o /dev/null -s -w 'connect=%{time_connect} tls=%{time_appconnect}\n' https://backend:9090/-/ready

# Detect MTU/fragmentation issues
ping -M do -s 1472 backend && echo "1500 MTU ok"

# Confirm the port speaks TLS
openssl s_client -connect backend:9090 -servername backend </dev/null

# Test the datasource via API
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" http://localhost:3000/api/datasources/uid/$UID/health | jq

Conclusion

net/http: TLS handshake timeout is a network-timing failure, not a trust failure: the TCP connection opened but TLS negotiation stalled. Confirm it’s timing (not x509) in the logs, time the handshake from the Grafana host, rule out MTU/fragmentation and silent packet drops, and only then adjust Grafana’s handshake timeouts. Most real cases trace to an overloaded backend or an MTU mismatch on a tunnel — fix the path and the timeouts vanish.