🚨 Telegraf Error Guide: 'panic: runtime error: nil pointer dereference' — Fix Telegraf Crashes

Overview

Most Telegraf problems are logged errors the agent recovers from. A panic is different: an unhandled Go runtime error crashes the entire process, taking every input and output down with it. A nil pointer dereference is the most common form, and it prints a goroutine stack trace before the process exits:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a3f5c1]

goroutine 214 [running]:
github.com/influxdata/telegraf/plugins/inputs/someplugin.(*Plugin).Gather(0xc00042a180, 0xc0003b6120)
        /go/src/github.com/influxdata/telegraf/plugins/inputs/someplugin/someplugin.go:212 +0x1c1
github.com/influxdata/telegraf/agent.(*Agent).gatherOnce(...)
        /go/src/github.com/influxdata/telegraf/agent/agent.go:451
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
        /go/src/github.com/influxdata/telegraf/agent/agent.go:389 +0x5a5

Under systemd the crash shows up as the service dying and (usually) restarting:

telegraf.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
telegraf.service: Failed with result 'exit-code'.

Unlike a logged E! error, a panic stops all collection until the offending plugin or input is removed or fixed.

Symptoms

• Telegraf exits entirely and systemctl status telegraf shows repeated restarts (a crash loop).
• The journal contains panic: followed by [signal SIGSEGV] and a goroutine ... [running] stack.
• All metrics — every input and output — stop, not just one plugin’s.
• The crash is reproducible on start-up or correlates with a specific device/endpoint returning malformed data.
• It began right after a config change, a plugin was enabled, or a version upgrade/downgrade.

Common Root Causes

• Plugin bug on malformed input — a plugin dereferences a field the remote system left null/empty (a device, API, or file with unexpected shape).
• Edge-case config — an option combination the plugin author did not guard, triggering a nil access at start-up.
• Version regression — a bug introduced in a specific Telegraf release; the same config was stable on the prior version.
• Custom/external execd plugin — a third-party or execd plugin panics and brings down the parent process.
• Corrupt or partial state — a stateful input reading a truncated file, cache, or checkpoint.
• Incompatible parser/data — a parser fed a payload shape it cannot handle, hitting a nil branch.

Diagnostic Workflow

First capture the full stack — the trace names the exact plugin and source line, which is the single most useful piece of evidence. Pull it from the journal without truncation:

journalctl -u telegraf --no-pager --since "-15min" | grep -A 40 'panic:'

The first plugin package in the stack (plugins/inputs/<name> or plugins/outputs/<name>) is the prime suspect. Bisect by running Telegraf with only that input to reproduce the crash in the foreground:

# Reproduce with a single input; a panic here confirms the culprit
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter someplugin --debug

If it panics, narrow further: comment out that [[inputs.someplugin]] block (or split its targets) to find the specific device/endpoint whose data triggers the nil dereference. You can also disable outputs while testing so you exercise only the input path:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter someplugin --output-filter '' --debug

To keep the rest of monitoring alive while you work, disable just the offending plugin. A per-plugin config file makes this surgical:

# Move the suspect plugin's config aside and restart
sudo mv /etc/telegraf/telegraf.d/someplugin.conf /root/someplugin.conf.disabled
sudo systemctl restart telegraf

Confirm the version, because a panic is frequently a known, already-fixed bug:

telegraf --version
# compare against the latest release / changelog before filing anything

If the panic persists on the latest version with a minimal config, it is a genuine upstream bug — capture the full stack, the redacted config block, and telegraf --version, and report it to the InfluxData GitHub issue tracker.

Example Root Cause Analysis

A monitoring host entered a crash loop: systemctl status telegraf showed it restarting every few seconds, and the journal held panic: runtime error: invalid memory address or nil pointer dereference with a stack pointing at plugins/inputs/snmp. Because a panic kills the whole agent, all metrics for the host had stopped, not just SNMP.

Running telegraf --test --input-filter snmp --debug reproduced the crash instantly. Splitting the agents list showed one newly added switch triggered it: that device returned a malformed SNMP table row the plugin did not guard against, and dereferencing a nil field crashed the process. As an immediate mitigation the team moved the SNMP config aside and restarted, restoring every other input. telegraf --version was two releases behind; the changelog listed a fix for exactly this nil dereference in the snmp input, and upgrading resolved it permanently. The lesson: a panic is all-or-nothing — isolate the plugin with --input-filter, disable it to restore the rest of collection, and check whether a newer release already fixed the bug before reporting upstream.

Prevention Best Practices

• Split inputs into per-plugin files under /etc/telegraf/telegraf.d/ so a single plugin can be disabled without editing a monolithic config.
• Pin and track Telegraf versions, and read the changelog for panic/crash fixes before and after upgrading.
• Test config changes with telegraf --test in a non-production environment before rolling out fleet-wide.
• Sandbox risky or third-party plugins behind execd where a crash is isolated to the child process, not the agent.
• Keep systemd Restart=on-failure with a RestartSec back-off so a crash loop does not hammer the host, and alert on restart counts.
• When you hit a genuine bug, file it upstream with the full stack, redacted config, and version so it gets fixed for everyone.

Quick Command Reference

# Capture the full panic stack from the journal
journalctl -u telegraf --no-pager --since "-15min" | grep -A 40 'panic:'

# Reproduce with a single input to confirm the culprit
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter someplugin --debug

# Disable the offending plugin and restart to restore the rest
sudo mv /etc/telegraf/telegraf.d/someplugin.conf /root/someplugin.conf.disabled
sudo systemctl restart telegraf

# Confirm version against the changelog
telegraf --version

# Watch for restart loops
systemctl status telegraf

Related Guides

• Telegraf Error: unknown input plugin
• Telegraf Error: invalid configuration
• Telegraf Error: [inputs.exec] exit status 1

More fixes in the Telegraf guides.

Conclusion

panic: runtime error: invalid memory address or nil pointer dereference crashes the entire Telegraf process, so every input and output stops until the offending plugin is removed or fixed. Capture the full goroutine stack — it names the exact plugin and line — then isolate it with --input-filter, disable that plugin to restore the rest of collection, and check whether your version already has a fix. Most panics are plugin bugs triggered by malformed input or a version regression, so upgrade first and report a minimal reproduction upstream when the crash survives the latest release.