Telegraf Error Guide: 'parsing metrics failed: invalid

Overview

The http input fetches a URL and hands the raw body to whichever parser data_format selects. When data_format = "json" but the endpoint returns an HTML page (a login redirect, a 500 error page, a proxy notice), the JSON parser chokes on the very first < character:

2026-07-12T12:00:00Z E! [inputs.http] Error in plugin: parsing metrics failed: invalid character '<' looking for beginning of value

A closely related variant appears when the body is JSON but the parser is pointed at the wrong sub-document or a scalar, so it finds a value where it expected an object:

E! [inputs.http] Error in plugin: parsing metrics failed: invalid character 'p' after top-level value

Telegraf keeps running, but the affected URL produces no metrics until the body actually matches the parser.

Symptoms

Metrics from one [[inputs.http]] URL are missing while other inputs collect fine.
journalctl -u telegraf repeats parsing metrics failed: invalid character '<' every interval.
Fetching the URL with curl returns HTML (<!DOCTYPE html> or <html>) rather than JSON.
The endpoint recently added authentication and now serves a login/redirect page to unauthenticated clients.
The JSON is valid but json_query or json_v2 paths return nothing or the wrong shape.

Common Root Causes

Endpoint returns HTML, not JSON — an error page, login redirect, or maintenance page; the body starts with <.
Missing authentication — the API now requires a token, so it serves a 401/302 HTML page instead of data.
Wrong data_format — the endpoint emits Prometheus or XML but the config says json.
Wrong json_query — the GJSON path points at a missing or scalar node, so no object is found to parse into fields.
Time field misconfigured — json_time_key/json_time_format do not match the payload, causing per-metric parse failures.
Should be json_v2 — deeply nested or array-heavy payloads need the json_v2 parser, not the flat json parser.
Content-encoding / proxy interception — a proxy injects an HTML notice or a gzip body is not decoded.

Diagnostic Workflow

Always look at the raw bytes first. If the body starts with <, it is not JSON and no parser tuning will help until the request returns data:

curl -sS -i https://api.example.com/metrics | head -n 20

Check the status code and Content-Type. A 302/401 with text/html means you are getting an error or login page. Add the auth the endpoint expects, then confirm the body is valid JSON:

curl -sS -H "Authorization: Bearer ${API_TOKEN}" https://api.example.com/metrics | jq . | head

Once the body is genuine JSON, run only the http input to confirm the parser extracts fields:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter http --debug

For a simple flat payload, the classic json parser with an explicit query and time key looks like this:

[[inputs.http]]
  urls = ["https://api.example.com/metrics"]
  data_format = "json"
  json_query = "data.stats"
  tag_keys = ["region"]
  json_time_key = "timestamp"
  json_time_format = "2006-01-02T15:04:05Z07:00"

  [inputs.http.headers]
    Authorization = "Bearer ${API_TOKEN}"
    Accept = "application/json"

For nested objects or arrays of readings, the json_v2 parser maps explicit paths and is far more predictable:

[[inputs.http]]
  urls = ["https://api.example.com/metrics"]
  data_format = "json_v2"

  [[inputs.http.json_v2]]
    measurement_name = "api_stats"
    [[inputs.http.json_v2.object]]
      path = "data.stats"
      timestamp_key = "timestamp"
      timestamp_format = "unix"
      tags = ["region"]

The Accept: application/json header matters: many APIs content-negotiate and will return HTML unless you ask for JSON explicitly.

Example Root Cause Analysis

An internal api_stats collection started logging parsing metrics failed: invalid character '<' looking for beginning of value right after a security rollout. The config had not changed, so the team suspected a Telegraf bug. Running curl -i against the URL told the real story: a 302 Found redirect to /login, with an HTML body — the API now required a bearer token and served the SSO login page to anyone without one.

The < the parser complained about was the < of <!DOCTYPE html>. Adding an Authorization: Bearer ${API_TOKEN} header (with the token supplied via the service environment) made the endpoint return JSON again, and telegraf --test immediately printed api_stats fields. The lesson: invalid character '<' is almost never a JSON problem — it means the response is HTML, so inspect the raw body and status code before touching parser options.

Prevention Best Practices

Send an explicit Accept: application/json header so content-negotiating endpoints never fall back to HTML.
Store API tokens in the service environment (${API_TOKEN}) and reference them in [inputs.http.headers], never inline.
Validate the payload shape with curl ... | jq . before writing json_query/json_v2 paths.
Prefer json_v2 for nested or array payloads; it fails loudly on the exact path rather than silently dropping fields.
Set json_time_key/json_time_format to match the payload precisely, or omit them to let Telegraf timestamp on collection.
Alert on a sudden drop to zero metrics for a URL — an auth change that starts serving HTML looks exactly like this.

Quick Command Reference

# Inspect status, headers, and the first bytes of the body
curl -sS -i https://api.example.com/metrics | head -n 20

# Confirm the body is valid JSON with auth applied
curl -sS -H "Authorization: Bearer ${API_TOKEN}" https://api.example.com/metrics | jq .

# Run only the http input with debug
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter http --debug

# Watch parse errors live
journalctl -u telegraf -f | grep -i 'parsing metrics failed'

More fixes in the Telegraf guides.

Conclusion

parsing metrics failed: invalid character '<' looking for beginning of value means the JSON parser was handed HTML — usually an error page or a login redirect because authentication is missing. Inspect the raw body and status code with curl -i first; if it starts with <, fix the request (add auth, Accept: application/json) rather than the parser. Once the endpoint truly returns JSON, use json_query for flat payloads or json_v2 for nested ones, and validate every path with jq before deploying.

Telegraf Error Guide: 'parsing metrics failed: invalid character' — Fix JSON Parse Errors

Fixing errors like this? Get 500 free DevOps AI prompts

Overview

Symptoms

Common Root Causes

Diagnostic Workflow

Example Root Cause Analysis

Prevention Best Practices

Quick Command Reference

Conclusion

Get 500 Battle-Tested DevOps AI Prompts — Free

Fixing errors like this? Get 500 free DevOps AI prompts

Overview

Symptoms

Common Root Causes

Diagnostic Workflow

Example Root Cause Analysis

Prevention Best Practices

Quick Command Reference

Related Guides

Conclusion

Get 500 Battle-Tested DevOps AI Prompts — Free