🚨 Logstash Error Guide: 'Received an event that has a different character encoding' — Fix the Codec Charset

Overview

Logstash treats every string internally as UTF-8. When bytes arrive on an input that are not valid UTF-8 — a Latin-1 accented character, a Windows-1252 smart quote, a UTF-16 file with a byte-order mark, or a truncated multibyte sequence — the codec that decodes those bytes cannot map them cleanly, and Logstash logs a warning as it force-converts them.

The message appears in /var/log/logstash/logstash-plain.log:

[2026-07-10T14:22:03,451][WARN ][logstash.codecs.line     ][main][a1b2...] Received an event that has a different character encoding than you configured. {:text=>"caf\\xE9 user login failed", :expected_charset=>"UTF-8"}

The same warning surfaces from other codecs with slightly different logger names — logstash.codecs.plain, logstash.codecs.json, or logstash.codecs.multiline:

[2026-07-10T14:22:04,002][WARN ][logstash.codecs.json     ][main] JSON parse error, original data now in message field {:message=>"Unexpected character ('\\xE9')", :exception=>LogStash::Json::ParserError}

What it means: the input’s codec has a charset setting (default UTF-8). It decodes the incoming byte stream using that charset, then re-encodes to UTF-8 for the rest of the pipeline. When the real bytes are in a different encoding than charset claims, the decode produces replacement characters (�, shown as ? or �) or leaves raw escape sequences like \xE9. The event is not dropped — it flows through mangled. This is a data-integrity problem, not a pipeline-halt: your message field ends up with mojibake, and any grok or json filter downstream may then fail to parse it.

Symptoms

• The logstash-plain.log warning Received an event that has a different character encoding than you configured, repeated once per affected line.
• Fields in Kibana render accented or non-Latin characters as � (U+FFFD replacement) or literal escapes like caf\xE9.
• A json codec or filter throws ParserError on lines that contain non-ASCII bytes, dropping the structured fields and dumping the raw string into message with a _jsoncodecfailure / _jsonparsefailure tag.
• Only some sources are affected — typically Windows hosts (Windows-1252), legacy apps writing Latin-1, or files exported as UTF-16.
• The very first event from a UTF-16 file carries a stray BOM () at the start of message.

Common Root Causes

• Source is not UTF-8 — the application writes Latin-1 (ISO-8859-1) or Windows-1252 (smart quotes, en-dashes, €), but the input codec’s charset is left at the UTF-8 default.
• UTF-16 files with a BOM — Windows PowerShell and some exporters write UTF-16LE with a byte-order mark; decoded as UTF-8 the whole file is garbage.
• Invalid / truncated multibyte sequences — a log line was split mid-character (log rotation, a partial network frame on a TCP input), leaving an incomplete UTF-8 byte sequence that cannot decode.
• Mixed encodings in one stream — several apps ship to the same input; some are UTF-8, some Latin-1. No single charset is correct for all of them.
• Binary or control bytes in the message — a stack trace, a hex dump, or an accidental binary blob written to a log file.
• Codec/charset mismatch between Beats and Logstash — Filebeat reads a file with its own encoding setting; if Filebeat already mis-decoded, Logstash inherits the damage regardless of its own charset.

Diagnostic Workflow

First, confirm the config is syntactically valid before you change codecs on a live node:

# Validate the whole configured pipeline
sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit \
  --path.settings /etc/logstash

# Validate just the file you are editing
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/app.conf --config.test_and_exit

Next, find out what encoding the source actually is — do not guess. Inspect the raw bytes on disk:

# What does 'file' think the encoding is?
file -i /var/log/legacy-app/app.log
#   /var/log/legacy-app/app.log: text/plain; charset=iso-8859-1

# Hunt for a UTF-16 BOM (FF FE) or non-UTF-8 bytes
head -c 64 /var/log/legacy-app/app.log | xxd | head

# List bytes that are not valid UTF-8
grep -aP '[\x80-\xFF]' /var/log/legacy-app/app.log | head

Once you know the real charset, set it explicitly on the input’s codec. Here is a realistic pipeline that handles a Latin-1 file input and a separate TCP input that receives Windows-1252:

input {
  # A legacy app that writes ISO-8859-1 to disk
  file {
    path     => "/var/log/legacy-app/app.log"
    codec    => plain { charset => "ISO-8859-1" }
    sincedb_path => "/var/lib/logstash/sincedb_legacy"
  }

  # A Windows service shipping over TCP in Windows-1252
  tcp {
    port  => 5045
    codec => line { charset => "CP1252" }
  }

  # UTF-16 exports (BOM-prefixed)
  file {
    path  => "/var/log/exports/*.log"
    codec => plain { charset => "UTF-16" }
  }
}

filter {
  # Strip a leftover BOM if one slips through
  mutate { gsub => [ "message", "", "" ] }

  json {
    source          => "message"
    skip_on_invalid_json => true
    tag_on_failure  => [ "_jsonparsefailure" ]
  }
}

output {
  stdout { codec => rubydebug }
}

Prove the decode is correct by feeding known bytes through a one-liner and watching rubydebug:

printf 'caf\xE9 login failed\n' | \
  bin/logstash -e 'input{stdin{codec=>plain{charset=>"ISO-8859-1"}}} output{stdout{codec=>rubydebug}}'
# message should read: "café login failed"  (not "caf\xE9" or "caf�")

Then tail the plain log to confirm the warnings stop after redeploy:

grep -i 'different character encoding\|\\xE9\|jsonparsefailure' /var/log/logstash/logstash-plain.log

Example Root Cause Analysis

An operations team saw thousands of Received an event that has a different character encoding warnings after onboarding a Windows-based billing service. In Kibana, customer names with accents (Müller, Renée) showed as M�ller and Ren�e, and a downstream json filter was tagging 5% of events with _jsonparsefailure.

Running file -i on the shipped log reported charset=iso-8859-1, but a closer look with xxd showed the byte 0x93 — a Windows-1252 “curly” opening quote that does not exist in pure ISO-8859-1. The source was actually Windows-1252, a superset of Latin-1. Their Logstash file input used the default UTF-8 codec, so every accented byte decoded to �, and the json filter then choked on lines whose quoted fields contained mangled characters.

The fix was two lines: set codec => plain { charset => "CP1252" } on that input and add skip_on_invalid_json => true to the json filter so a genuinely malformed line degrades gracefully into message instead of failing. They also discovered a second, subtler source — a nightly UTF-16 export from a reporting tool — whose first event carried a  BOM; a mutate { gsub => [ "message", "", "" ] } removed it. After redeploying, the encoding warnings and the JSON parse failures both went to zero, and Müller rendered correctly.

Prevention Best Practices

• Fix encoding at the source where possible — configure the application or logging framework to emit UTF-8. Every downstream fix is a workaround.
• Set charset explicitly on any input reading a non-UTF-8 source; never rely on the UTF-8 default for legacy or Windows systems.
• Use CP1252, not ISO-8859-1, for Windows sources. Windows-1252 is a superset — using Latin-1 silently mangles smart quotes, dashes, and the euro sign.
• Set Filebeat’s encoding to match the file when Beats does the reading; if Filebeat mis-decodes, Logstash cannot recover the original bytes.
• Strip BOMs defensively with mutate gsub on  when any UTF-16/UTF-8-BOM source is possible.
• Make JSON parsing forgiving with skip_on_invalid_json => true and a tag_on_failure, so one bad byte does not discard an event’s structure silently.
• Alert on the warning — a sudden burst of different character encoding in logstash-plain.log usually means a new source shipped with an unexpected charset.

Quick Command Reference

# Validate pipeline syntax
bin/logstash --config.test_and_exit --path.settings /etc/logstash

# Detect the real charset of a log file
file -i /var/log/legacy-app/app.log

# Look for a UTF-16 BOM or non-UTF-8 bytes
head -c 64 /var/log/legacy-app/app.log | xxd | head
grep -aP '[\x80-\xFF]' /var/log/legacy-app/app.log | head

# Test a charset decode end-to-end
printf 'caf\xE9\n' | bin/logstash -e \
  'input{stdin{codec=>plain{charset=>"ISO-8859-1"}}} output{stdout{codec=>rubydebug}}'

# Convert a file to UTF-8 out-of-band if you must
iconv -f WINDOWS-1252 -t UTF-8 in.log -o out.log

# Watch for encoding warnings after redeploy
grep -i 'different character encoding' /var/log/logstash/logstash-plain.log

Conclusion

Received an event that has a different character encoding than you configured means the bytes on the wire do not match the charset your input codec assumed. Logstash does not drop the event — it force-converts it, leaving � replacement characters or raw escapes that then break downstream json and grok filters. The reliable fix is to identify the real encoding with file -i and xxd, then set charset explicitly on the input codec (CP1252 for Windows, ISO-8859-1 for Latin-1, UTF-16 for BOM-prefixed exports), strip stray BOMs, and make JSON parsing forgiving. Validate with --config.test_and_exit, prove the decode through a stdin/rubydebug one-liner, and fix encoding at the source whenever you can. More parsing and codec fixes live in the Logstash category.