🚨 Grafana Error Guide: 'Live channel connection failed' — Fix the WebSocket Proxy Path

Overview

Grafana Live powers real-time features — streaming panels, dashboard-save notifications, alerting state pushes, and the “someone else is editing” banner — over a persistent WebSocket. When the browser cannot establish or hold that socket, the UI logs a repeating error and live features silently stop updating:

Live: connection failed. WebSocket connection to 'wss://grafana.example.com/api/live/ws' failed

The server side of the same failure usually appears in the Grafana log as an upgrade that never completes:

logger=live.gateway t=2026-07-08T10:14:22Z level=warn msg="error upgrading connection" error="websocket: request origin not allowed by Upgrader.CheckOrigin"

The defining symptom: the rest of Grafana works, but anything real-time (streaming data, live edit locks) is dead, and the browser console shows the wss://.../api/live/ws handshake failing on repeat.

Symptoms

• Browser console repeats WebSocket connection to 'wss://.../api/live/ws' failed.
• Streaming panels and the Explore live tail never update; the dashboard “was saved” toast from other editors never arrives.
• Grafana log shows error upgrading connection or request origin not allowed.
• curl to /api/health succeeds but the /api/live/ws upgrade returns 400, 403, or 502.
• The failure appeared right after putting Grafana behind Nginx, an ALB/ingress, or Cloudflare.

Common Root Causes

• Reverse proxy strips the Upgrade headers — the proxy does not pass Upgrade: websocket / Connection: upgrade, so the handshake is downgraded to a plain HTTP request that Grafana rejects.
• root_url mismatch — Grafana’s configured root_url does not match the external URL the browser used, so the origin check on the WebSocket fails.
• allow_embedding / origin policy — when Grafana is embedded or served from a different host than expected, the Live upgrader’s origin check blocks the connection.
• HTTP/1.1 not preserved — the proxy talks HTTP/1.0 or buffers, and WebSocket upgrade requires HTTP/1.1.
• Idle timeout too short — a load balancer closes the idle socket after 30–60s, so the connection flaps constantly.
• Connection/user limit reached — [live] max_connections is exhausted on a busy instance and new sockets are refused.

Diagnostic Workflow

Reproduce the upgrade directly against Grafana, bypassing the proxy, to isolate whether Grafana itself accepts the WebSocket:

# From the Grafana host, confirm the upgrade returns 101 Switching Protocols
curl -i -N \
  -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" \
  -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  http://localhost:3000/api/live/ws

A healthy backend answers HTTP/1.1 101 Switching Protocols. If direct works but through the proxy fails, the proxy is the problem.

Check the two Grafana settings that govern the external URL and the origin check in grafana.ini:

[server]
# Must equal the exact URL the browser uses, including scheme and any subpath
root_url = https://grafana.example.com/

[security]
# Required when Grafana is embedded or served cross-origin
allow_embedding = true

[live]
# Raise if you exhaust sockets on a busy multi-user instance
max_connections = 100

Watch the Live gateway log while you reload the dashboard:

grep -i 'live' /var/log/grafana/grafana.log | tail -n 20
journalctl -u grafana-server --since '5 min ago' | grep -i 'upgrad\|origin\|live'

For Nginx, the upgrade headers and HTTP/1.1 must be set explicitly on the Grafana location:

location / {
    proxy_pass http://grafana_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 3600s;   # keep the socket open longer than the client ping
}

On an AWS ALB, confirm the target group uses a protocol version that supports WebSocket and raise the idle timeout above Grafana’s heartbeat; on a Kubernetes ingress-nginx, the same behavior is enabled via annotations for read/send timeouts.

Example Root Cause Analysis

A team moved Grafana behind ingress-nginx and users reported that streaming dashboards froze while everything else worked. The browser console showed WebSocket connection to 'wss://grafana.example.com/api/live/ws' failed every few seconds, and the Grafana log had no error upgrading connection line at all — the request was not even reaching the Live gateway as an upgrade.

Running the direct curl upgrade against localhost:3000/api/live/ws returned 101 Switching Protocols, proving Grafana was fine. The ingress was terminating the connection because its default proxy-read-timeout of 60s was shorter than the client ping interval, so the socket was torn down and re-dialed in a loop. Raising the read and send timeouts on the ingress to 3600s stopped the flapping, and streaming panels resumed updating immediately. The root_url was already correct, so no origin change was needed — the fix was entirely in the proxy timeout.

Prevention Best Practices

• Templatize the reverse-proxy config so every Grafana deployment ships with proxy_http_version 1.1, the Upgrade/Connection headers, and a read timeout well above the Live heartbeat.
• Keep root_url in grafana.ini exactly equal to the external URL (scheme, host, and subpath) in every environment.
• Set load-balancer idle timeouts to several minutes so a normally quiet socket is never reaped as “idle.”
• Add a synthetic check that opens /api/live/ws and asserts 101, so a proxy regression is caught before users notice frozen panels.
• Size [live] max_connections for peak concurrent users plus headroom, and alert on live connection rejections.

Quick Command Reference

# Direct upgrade test (expect: HTTP/1.1 101 Switching Protocols)
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  http://localhost:3000/api/live/ws

# Confirm configured external URL matches the browser URL
grep -E '^\s*root_url' /etc/grafana/grafana.ini

# Tail Live gateway upgrade/origin errors
journalctl -u grafana-server --since '5 min ago' | grep -i 'upgrad\|origin\|live'

# Verify the plain UI is healthy (isolates Live from a general outage)
curl -s http://localhost:3000/api/health

Conclusion

Live channel connection failed / WebSocket connection to /api/live/ws failed almost never means Grafana itself is broken — it means the WebSocket upgrade is being dropped, downgraded, origin-rejected, or timed out before it reaches (or while it holds) the Live gateway. Prove where the break is with a direct curl upgrade against localhost:3000/api/live/ws: if that returns 101, fix the proxy (preserve HTTP/1.1 and the Upgrade headers, raise the idle timeout); if it fails, fix root_url/allow_embedding or raise [live] max_connections. Bake the proxy settings and a synthetic upgrade check into your deployment so real-time features stay real-time.