🚨 OpenTelemetry Error Guide: 'instrument has exceeded the maximum allowed cardinality' — Fix Metric Overflow

Overview

To protect the process from unbounded memory growth, the OpenTelemetry metrics SDK caps the number of distinct attribute sets (streams) per instrument. When an instrument exceeds that cap, the SDK logs a warning and folds further data into a single overflow stream:

Warning: Instrument http.server.duration has exceeded the maximum allowed cardinality (2000). Discarding new attribute sets into the overflow attribute set {otel.metric.overflow="true"}.

The overflow appears in the backend as a catch-all series that hides real dimensions:

http_server_duration_bucket{otel_metric_overflow="true", le="+Inf"} 41235

Data isn’t dropped outright, but every stream past the limit is merged into one indistinct bucket, making the metric useless for its intended breakdown.

Symptoms

• SDK logs repeat exceeded the maximum allowed cardinality for a specific instrument.
• The backend shows an otel.metric.overflow="true" series absorbing most traffic.
• Dashboards lose per-route/per-tenant detail; only the overflow series grows.
• Process memory for the metrics SDK climbs before the limit kicks in.
• Started after adding an attribute like user ID, full URL path, or request ID to a metric.

Common Root Causes

• Unbounded attribute value — user/session/request IDs, raw URLs, or trace IDs used as metric attributes.
• Full path instead of route template — /orders/12345 recorded instead of /orders/{id}.
• Exact status/error strings — free-form error messages as an attribute value.
• Per-instance labels on shared metrics — host, pod, or IP where a bucketed dimension would do.
• Attribute leakage from auto-instrumentation — a library adds a high-cardinality attribute by default.

Diagnostic Workflow

Identify the instrument from the warning, then find which attribute is unbounded by inspecting the emitted metric locally with the debug exporter or a Prometheus scrape:

export OTEL_METRICS_EXPORTER=console   # or prometheus, then scrape :<port>/metrics
# Look for the attribute whose value changes on nearly every request

Fix it at the SDK with a View that drops or keeps only bounded attributes for that instrument. Python example:

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View

view = View(
    instrument_name="http.server.duration",
    # Keep only bounded keys; drop high-cardinality ones like user_id / full path
    attribute_keys={"http.request.method", "http.route", "http.response.status_code"},
)
provider = MeterProvider(views=[view])

Replace unbounded values with bounded ones before recording — use the route template, not the raw path:

# Bad: full path is unbounded
histogram.record(dur, {"http.target": "/orders/12345"})
# Good: templated route is bounded
histogram.record(dur, {"http.route": "/orders/{id}"})

If the dimensions are legitimately needed and bounded, raise the per-instrument cardinality limit via environment configuration instead of silently overflowing:

export OTEL_GO_X_CARDINALITY_LIMIT=5000   # SDK-specific; confirm your language's knob

Confirm the overflow series stops growing after the change by re-scraping and checking for otel.metric.overflow.

Example Root Cause Analysis

A service recorded http.server.duration with an http.target attribute set to the full request path. Because paths embedded order IDs (/orders/91integer), each request produced a new attribute set, and within an hour the SDK logged exceeded the maximum allowed cardinality (2000) and began folding everything into otel.metric.overflow="true". The latency dashboard flatlined into one meaningless line.

The fix used a View to keep only http.request.method, http.route, and http.response.status_code, and the instrumentation was changed to record the route template instead of the raw path. Distinct streams dropped from thousands to a few dozen, the overflow series disappeared, and the per-route latency breakdown returned. Memory use for the metrics SDK also fell noticeably.

Prevention Best Practices

• Never put unbounded values (user/session/request IDs, raw paths, trace IDs) directly on metric attributes; bucket or template them first.
• Define Views up front to pin the allowed attribute keys for high-traffic instruments, especially auto-instrumented HTTP/DB metrics.
• Record route templates (/orders/{id}) rather than concrete paths, and status classes where exact codes aren’t needed.
• Estimate series = product of attribute cardinalities before shipping a new dimension; keep it within the backend budget.
• Alert on the otel.metric.overflow series so a cardinality regression is caught the moment it appears.

Quick Command Reference

# Inspect emitted metrics locally to find the offending attribute
export OTEL_METRICS_EXPORTER=console

# Watch SDK logs for the warning
# (application logs) grep -i 'maximum allowed cardinality'

# Check the backend for the overflow series
# up{otel_metric_overflow="true"}  or  {otel.metric.overflow="true"}

# Raise the limit only for legitimately-bounded high dimensions (SDK-specific)
export OTEL_GO_X_CARDINALITY_LIMIT=5000

Conclusion

instrument has exceeded the maximum allowed cardinality means an instrument produced more distinct attribute sets than the SDK’s protective cap, so new data collapses into an overflow stream. The right fix is almost always to bound the attributes — drop unbounded keys with a View and record templated values — not to blindly raise the limit. Watching the overflow series keeps cardinality regressions from silently gutting your dashboards.