🚨 Prometheus Error Guide: 'HTTP response to HTTPS client' — Fix Scheme

Overview

Prometheus records this on the target status page when it opens a TLS handshake against an endpoint that is actually serving plain HTTP:

Get "https://10.0.0.7:9100/metrics": http: server gave HTTP response to HTTPS client

It is the mirror image of the opposite mistake, where a plain-HTTP client hits a TLS port and gets binary garbage or a handshake error. In both cases the scrape scheme does not match what the target speaks. The target shows DOWN, up{} is 0, and no samples are ingested until the scheme is aligned.

Symptoms

• Target DOWN with http: server gave HTTP response to HTTPS client.
• up{job="..."} == 0 only for jobs configured with scheme: https or a tls_config.
• curl https://target/metrics fails the handshake while curl http://target/metrics returns metrics.
• The failure appeared right after enabling TLS, a ServiceMonitor change, or a relabel that set __scheme__.

Common Root Causes

• scheme: https on a plain-HTTP exporter — most exporters (node_exporter, kube-state-metrics) serve HTTP by default.
• Relabeling set __scheme__ to https via a prometheus.io/scheme annotation or a discovery default that doesn’t match reality.
• ServiceMonitor scheme: https without the exporter’s web-config actually enabling TLS.
• Port confusion — the HTTPS port points at a container listening only on HTTP.
• TLS terminated elsewhere — a sidecar/proxy expected to do TLS isn’t in the path, so Prometheus talks TLS directly to the plaintext app.

Diagnostic Workflow

Confirm what the endpoint actually speaks by probing both schemes from the Prometheus host:

curl -sS -o /dev/null -w 'http=%{http_code}\n' http://10.0.0.7:9100/metrics
curl -skS -o /dev/null -w 'https=%{http_code}\n' https://10.0.0.7:9100/metrics

If HTTP works and HTTPS errors, the target is plaintext and the scrape scheme is wrong. Inspect the derived scheme on the Service Discovery page, then check the scrape config:

scrape_configs:
  - job_name: node
    scheme: http          # must match what the exporter serves
    static_configs:
      - targets: ["10.0.0.7:9100"]

For discovery-driven jobs, find any relabel rule setting the scheme:

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__     # this can force https incorrectly

For the Prometheus Operator, check the ServiceMonitor endpoint:

kubectl get servicemonitor my-app -o yaml | grep -A3 'scheme:'

Example Root Cause Analysis

A team enabled a cluster-wide default annotation prometheus.io/scheme: https intending to secure app traffic, but their node_exporter DaemonSet still served plaintext on :9100. Every node target flipped to DOWN with server gave HTTP response to HTTPS client.

Probing confirmed http=200 and the HTTPS probe failed the handshake. Because node_exporter was intentionally plaintext behind network policy, the correct fix was to scope the https default to app pods only and leave the node_exporter job on scheme: http. An alternative — actually enabling node_exporter’s --web.config.file TLS — was rejected as unnecessary for that trust boundary. Targets recovered immediately.

Prevention Best Practices

• Set scheme explicitly per job to match the exporter, rather than relying on a broad discovery default.
• When enabling scheme: https, enable TLS on the target in the same change (web-config or sidecar), never one without the other.
• Scope scheme-setting annotations narrowly; a cluster-wide prometheus.io/scheme breaks plaintext exporters.
• After any TLS or ServiceMonitor change, verify the target on the Service Discovery page shows the intended __scheme__.
• Probe both schemes with curl before flipping a job to https.

Quick Command Reference

# Which scheme does the target actually speak?
curl -sS  -o /dev/null -w 'http=%{http_code}\n'  http://TARGET:PORT/metrics
curl -skS -o /dev/null -w 'https=%{http_code}\n' https://TARGET:PORT/metrics

# Operator: inspect the ServiceMonitor scheme
kubectl get servicemonitor NAME -o yaml | grep -A3 'scheme:'

up == 0

Conclusion

server gave HTTP response to HTTPS client means Prometheus is speaking TLS to a plaintext endpoint. Probe both schemes to learn what the target actually serves, then align scheme (and any __scheme__ relabel or ServiceMonitor setting) to match — or enable TLS on the target if security requires it. Keep scheme defaults narrowly scoped so one annotation can’t break every plaintext exporter.