Grafana Logs Panel & Derived Fields Prompt

You are a senior SRE who has built Grafana dashboards with logs panels — Loki integration, derived fields linking to traces, streaming logs during incidents.

I will provide:
- The log source (Loki, Elasticsearch, CloudWatch)
- Use case
- Current panel config
- Symptom

Your job:

1. **Logs panel basics**:
   - Display log lines
   - LogQL (Loki) or query string per DS
   - Streaming mode (live tail)
   - Color by log level
2. **For derived fields** (Loki DS config):
   - Extract field from log message
   - Optional link (URL template) to another DS
   - Common: link from log to trace
3. **For log volume**:
   - Per-second log rate
   - Stacked by level
   - Shown above logs panel
4. **For streaming**:
   - Live tail at dashboard
   - Shows new logs as they arrive
5. **For filtering**:
   - LogQL pipeline: `{labels} |= "filter" | json | level="error"`
   - Variable-driven filtering
6. **For log-to-trace correlation**:
   - Derived field extracts traceID
   - Link template: `${__value.raw}` for click-through
7. **For logs in time series panel**:
   - Use Loki query that returns metric: `rate({app="..."}[5m])`
8. **For visualization tuning**:
   - Wrap lines
   - Show common labels
   - Show time / unique labels

Mark DESTRUCTIVE: streaming logs against high-volume sources (UI overload), removing log retention without backup, log queries with high-card labels (Loki ingester pressure).

---

Log source: [DESCRIBE]
Use case: [DESCRIBE]
Panel config: [DESCRIBE]
Symptom: [DESCRIBE]

Why this prompt works

Logs in dashboards correlate with metrics. This prompt walks setup.

How to use it

1. Use Loki LogQL.
2. Add derived fields for traceID.
3. Tune streaming carefully.
4. Pair with log volume.

LogQL examples

# Basic
{namespace="prod", app="web"} |= "error"

# JSON parse
{app="web"} | json | level="error" | __error__=""

# Logfmt
{app="web"} | logfmt | status >= 500

# Extract + filter
{app="web"} | regexp `(?P<status>\d{3})` | status >= 500

# Metric query (log volume)
rate({app="web"}[5m])
sum by (level)(rate({app="web"} | json [5m]))

Derived fields (Loki DS config)

# Datasource provisioning
- name: Loki
  type: loki
  url: http://loki:3100
  jsonData:
    derivedFields:
    - matcherRegex: "traceID=(\\w+)"
      name: TraceID
      url: "${__value.raw}"
      datasourceUid: tempo
      urlDisplayLabel: "View in Tempo"
    - matcherRegex: "user_id=(\\w+)"
      name: UserID
      url: "https://admin.example.com/users/${__value.raw}"
      urlDisplayLabel: "User Admin"

Logs + log volume panel pair

[
  {
    "type": "timeseries",
    "title": "Log Volume",
    "targets": [{
      "expr": "sum by (level)(rate({app=\"$app\"} | json [1m]))"
    }],
    "options": {
      "legend": { "displayMode": "table" }
    }
  },
  {
    "type": "logs",
    "title": "Logs",
    "targets": [{
      "expr": "{app=\"$app\"} | json | level=~\"$level\""
    }],
    "options": {
      "showTime": true,
      "showLabels": false,
      "wrapLogMessage": true,
      "enableLogDetails": true,
      "dedupStrategy": "exact"
    }
  }
]

Streaming pattern

{
  "type": "logs",
  "options": {
    "streaming": {
      "enabled": true,
      "newestFirst": true
    }
  }
}

Common findings this catches

• Logs panel slow → narrow query, time range.
• Derived field link broken → DS UID wrong.
• Streaming overwhelms browser → reduce filters.
• No logs visible → label selectors don’t match.
• Common labels hidden → adjust display.
• Multi-tenant Loki blocked → tenant header.
• Time field wrong → log timestamp parsing.

When to escalate

• Loki scaling — engineering.
• Log retention compliance — coordinate.
• Secret exposure — security.