🚨 VictoriaMetrics Error: 'duplicate time series on the right side of the operation' — Cause, Fix, and Troubleshooting Guide

Overview

When a MetricsQL binary operation (like +, /, or and) joins two vectors, vmselect (or single-node victoria-metrics) matches series on the left against series on the right by their label sets. Each series on one side must map to at most one series on the other. If several series on the right collapse to the same matching identity, the engine cannot decide which one to use and fails the query:

cannot execute query: duplicate time series on the right side of the operation: {instance="a"}; consider adding `on()` or `ignoring()` modifiers to the operation

This is a vector-matching problem, not a data or storage problem. It means the two sides of your operation do not have a clean one-to-one (or explicitly declared many-to-one) relationship. The fix is to tell MetricsQL exactly which labels define the match, or to collapse the duplicates first.

Symptoms

• A previously working panel or recording rule breaks with duplicate time series on the right side of the operation.
• The error names a small label set (for example {instance="a"}) that appears on more than one series after matching.
• A division or ratio expression fails, but each half of it returns valid results on its own.
• Adding a new exporter, relabeling rule, or extra label to a metric suddenly triggers the error on unrelated queries that join that metric.
• The same expression works in one environment and fails in another where a metric carries an extra label.

Common Root Causes

• No matching modifier on a binary op — two vectors with different label sets joined directly, so MetricsQL cannot line them up one-to-one.
• Missing group_left / group_right for many-to-one — you are joining a per-pod metric against a per-node metric without declaring which side is the “many”.
• A label leak creating duplicate identity — a stray label (or a dropped one) makes several distinct series reduce to the same match key on the right side.
• Aggregation that leaves duplicates — an aggregated result still has more than one series sharing the labels used for matching.

How to diagnose

Run each side of the operation separately and look at the label sets that survive matching. Start by listing the right-hand series:

curl -s -G 'http://localhost:8481/select/0/prometheus/api/v1/query' \
  --data-urlencode 'query=<right_hand_side_expression>' \
  | python3 -c 'import sys,json;[print(s["metric"]) for s in json.load(sys.stdin)["data"]["result"]]'

Count how many series share the labels you intend to match on. If count by(instance)(...) returns anything greater than 1 for a value, that instance is your duplicate:

count by(instance) (<right_hand_side_expression>)

Reproduce and confirm which extra label is causing the collision by inspecting full label sets in vmui (http://localhost:8481 in a cluster, http://localhost:8428 single-node). Compare the labels on the left and right — the ones that differ are what you must scope with on() or drop with ignoring().

Fixes

1. Scope the match with on(...) or ignoring(...). Tell MetricsQL exactly which labels define equality so the extra labels no longer split or duplicate the match:

# Match only on instance, ignore every other label:
rate(http_requests_total[5m]) / on(instance) node_cpu_count

Use ignoring(...) instead when it is easier to name the labels to exclude:

rate(http_requests_total[5m]) / ignoring(handler, method) node_cpu_count

2. Declare a many-to-one join with group_left / group_right. When one side legitimately has many series per matching key (for example many pods per node), tell MetricsQL which side is the “many”:

# Many pod-level series joined to one node-level series:
container_memory_usage_bytes
  * on(node) group_left(role) node_role_info

Use group_right(...) when the “many” side is on the right.

3. Collapse the duplicates with an aggregation. If the right side genuinely should be a single series per key, reduce it before the operation:

rate(http_requests_total[5m]) / on(instance) max by(instance)(node_cpu_count)

Pick the aggregation (max, sum, avg) that reflects the intended semantics.

4. Fix the underlying label leak. If a stray label is what makes series collide on the right, drop or trim it at ingest with metric_relabel_configs, or exclude it in the query, so the match key is unique again.

What to watch out for

• on() and ignoring() are opposites — on() lists the only labels to match, ignoring() lists the labels to drop from an otherwise full match; mixing up which you need is a common trap.
• group_left / group_right are for many-to-one and one-to-many joins, not a blanket silencer — using them to paper over an unexpected duplicate hides a real cardinality or relabeling bug.
• Aggregating to remove duplicates changes the meaning of the result; make sure max/sum/avg is actually what you want, not just what makes the error disappear.
• The error can surface long after the join was written, the moment a new label is added upstream — treat a sudden appearance as a signal to audit recent relabeling changes.

Related

• VictoriaMetrics Error Guide: cannot parse MetricsQL
• VictoriaMetrics Error Guide: max series
• VictoriaMetrics Error Guide: high churn rate