GitLab Webhooks & External Triggers Debug Prompt

You are a senior DevOps engineer who has wired up GitLab webhooks to Slack, Jira, custom services, and SIEM. You know how to diagnose silent delivery failures and the difference between project, group, and system hooks.

I will provide:
- The symptom (webhook not firing, fired but no response, signature verification failing, duplicate deliveries)
- The webhook config (URL, events selected, SSL verification, custom headers)
- Recent delivery history (in GitLab UI: Settings → Webhooks → click webhook → Recent events)
- Receiver side: logs from the service that should receive the webhook

Your job:

1. **Verify the webhook is registered**:
   - Project webhooks: Settings → Webhooks
   - Group webhooks (Premium): Group → Settings → Webhooks
   - System webhooks (admin): Admin → System hooks
2. **For "webhook not firing"**:
   - Event selection correct? (Push events, Tag events, MR events, etc.)
   - URL reachable from GitLab? Test with `curl` from GitLab host (or a debug pod)
   - Webhook in "Recent events" shows attempts? Failed attempts include status code and response
3. **For "webhook fires but no action"**:
   - Receiver returning 200 OK but not processing? Check receiver logs
   - JSON body malformed for receiver's expectations
   - Webhook signature (`X-Gitlab-Token` header) — receiver should validate this matches the secret
4. **For 5xx / connection errors**:
   - GitLab side: 10s default timeout — slow receivers cause retries
   - SSL verification failures: receiver cert issues, or GitLab admin disabled "Allow self-signed"
   - Connection refused: receiver down or firewall blocks GitLab's IP
5. **Pipeline triggers (`POST /api/v4/projects/:id/trigger/pipeline`)**:
   - Pipeline trigger tokens: Settings → CI/CD → Pipeline triggers → Add token
   - Use `token` as form data; `ref` for branch/tag; `variables[VAR]=value` for vars
   - Triggered pipelines have `$CI_PIPELINE_SOURCE=trigger`
6. **For external service calling GitLab API**:
   - Use a personal access token, project access token, or trigger token (scope-aware)
   - Trigger tokens are project-scoped and can only create pipelines
   - PAT/PrAT have broader scopes
7. **For securing webhooks**:
   - Always set the webhook `Secret token` — receiver validates header
   - HTTPS only (GitLab default; admin can allow HTTP)
   - Restrict events to what the receiver actually needs
   - Don't include sensitive data in custom URL templates
8. **For duplicate deliveries**:
   - GitLab retries on non-2xx responses (up to N times)
   - If receiver returns 5xx but actually processed, retries cause duplicate work
   - Receiver should be idempotent (use delivery ID for dedup)

Mark DESTRUCTIVE: leaking webhook secrets via accidental log inclusion, deleting a webhook that's the only path to an external system, allowing all events without rate limiting (DoS receiver).

---

Symptom: [DESCRIBE]
Webhook config:
```
[DESCRIBE — URL, events, secret status (don't paste secret)]
```
Recent delivery history:
```
[PASTE — status codes and timestamps from "Recent events"]
```
Receiver-side logs:
```
[PASTE]
```

Why this prompt works

Webhook debugging is split-system: GitLab’s delivery view shows attempts; the receiver’s logs show processing. Most failures look opaque without both. This prompt structures the diagnosis.

How to use it

1. Always check both sides: GitLab “Recent events” AND receiver logs.
2. For signature issues, confirm both sides know the same secret.
3. For triggers, distinguish webhook (incoming to receiver) from trigger token (incoming to GitLab).
4. Test with curl from the same network GitLab uses, to rule out connectivity.

Useful commands

# Test webhook delivery manually
curl -X POST -H "X-Gitlab-Token: <secret>" -H "Content-Type: application/json" \
    -d '{"object_kind":"push","ref":"refs/heads/main"}' \
    https://receiver.example.com/webhook

# Trigger a pipeline via API
curl --request POST \
    --form "token=$TRIGGER_TOKEN" \
    --form "ref=main" \
    --form "variables[FOO]=bar" \
    "https://gitlab.example.com/api/v4/projects/<id>/trigger/pipeline"

# List webhooks via API
curl --header "PRIVATE-TOKEN: <t>" \
    "https://gitlab.example.com/api/v4/projects/<id>/hooks" | jq

# Update webhook
curl --request PUT --header "PRIVATE-TOKEN: <t>" --header "Content-Type: application/json" \
    --data '{"url":"https://new-receiver.example.com/webhook","push_events":true,"token":"newsecret"}' \
    "https://gitlab.example.com/api/v4/projects/<id>/hooks/<hook-id>"

# View recent delivery attempts (UI only; no API)
# Settings → Webhooks → click webhook → Recent events

Receiver verification pattern (Node.js example)

import express from 'express';
import crypto from 'crypto';

const app = express();
app.use(express.json({ limit: '10mb' }));

const WEBHOOK_SECRET = process.env.GITLAB_WEBHOOK_SECRET;

app.post('/webhook', (req, res) => {
  // Verify signature
  const signature = req.headers['x-gitlab-token'];
  if (signature !== WEBHOOK_SECRET) {
    return res.status(401).send('Invalid signature');
  }

  // Acknowledge fast
  res.status(200).send('OK');

  // Process async to avoid timeout
  setImmediate(() => processEvent(req.body, req.headers));
});

function processEvent(body, headers) {
  const eventType = headers['x-gitlab-event'];
  console.log(`Received ${eventType}: ${body.object_kind || 'unknown'}`);
  // ... actual handling
}

app.listen(3000);

Event types reference

Event	GitLab term	Useful for
Push	object_kind: push	CI triggers, mirror sync
Tag push	object_kind: tag_push	Release pipelines
Merge request	object_kind: merge_request	Review tooling, code review bots
Issue	object_kind: issue	Sync to Jira, notification
Note (comment)	object_kind: note	Chat bots, /commands
Pipeline	object_kind: pipeline	Notification, status sync
Job	object_kind: build	Per-job metrics
Deployment	object_kind: deployment	Notify DORA tracker
Release	object_kind: release	Changelog auto-publish

Common findings this catches

• Webhook URL returns 405 Method Not Allowed → receiver expects GET but webhook is POST.
• Signature mismatch → secret in webhook config doesn’t match receiver’s expected value.
• Receiver returns 200 but processes nothing → JSON parsing error in receiver; log raw body.
• GitLab retries 5x → receiver intermittent failures; verify TLS chain, response time.
• Webhook fires on push but branch: filter wrong → most webhooks fire for all branches; filter in receiver.
• Trigger token works for main but not protected branches → trigger token’s user lacks protected branch access.
• Webhook fires on every commit (monorepo) → stampede; debounce in receiver or filter event types.

Pipeline trigger from external

# Generate trigger token: Settings → CI/CD → Pipeline triggers
# In external system (e.g., nightly cron job):

curl --request POST \
    --form "token=$TRIGGER_TOKEN" \
    --form "ref=main" \
    --form "variables[ENV]=production" \
    --form "variables[VERSION]=$(date +%F)" \
    "https://gitlab.example.com/api/v4/projects/<id>/trigger/pipeline"

# Response includes pipeline URL + ID:
# {"id":123,"web_url":"https://...","status":"pending"}

When to escalate

• Persistent delivery failures across many webhooks — GitLab-side issue; check admin panel for webhook errors.
• Receiver compromise from spoofed events — rotate secret, audit logs.
• Large event volume overwhelming receiver — implement event filtering at GitLab side or aggregator pattern.