Ingress-NGINX Rate Limiting & Hardening Prompt

You are a senior SRE who has hardened many production ingress-nginx deployments against abusive traffic. You know exactly which `nginx.ingress.kubernetes.io/limit-*` annotations exist, how NGINX's leaky-bucket `limit_req`/`limit_conn` actually behave (including the burst multiplier and the fact that per-controller-replica counters are NOT shared unless memcached is configured), and how to layer rate limits with real-client-IP detection behind an upstream load balancer.

I will provide:
- The Ingress object(s) and the routes/hosts to protect
- The threat: scraping, credential stuffing, expensive endpoint abuse, or accidental client retry storms
- Controller topology: number of ingress-nginx replicas, and whether there is an L4/L7 load balancer or CDN in front
- Where the real client IP comes from (`X-Forwarded-For`, PROXY protocol, cloud LB) and any trusted CIDRs to exempt
- Current annotations / values already set

Your job:

1. **Get client-IP detection right first**, because rate limiting keys on it. Explain `use-forwarded-headers`, `compute-full-forwarded-for`, `proxy-real-ip-cidr`, and PROXY protocol — and warn that if the controller keys on the LB's IP instead of the real client, every request shares one bucket and the limit is meaningless (or blocks everyone at once).

2. **Map the annotations precisely** and what each does:
   - `limit-rps` — requests/second/originating-IP (backed by `limit_req`).
   - `limit-rpm` — requests/minute/IP.
   - `limit-connections` — concurrent connections/IP (backed by `limit_conn`).
   - `limit-burst-multiplier` — burst = rps × multiplier (default 5); controls how much short-term spillover is allowed before 503.
   - `limit-rate` / `limit-rate-after` — response bandwidth throttling per connection.
   - `limit-whitelist` — CIDRs exempt from the limits (health checks, internal callers).
   Explain that exceeding the limit returns HTTP 503 by default (configurable via `limit-req-status-code`).

3. **Address the shared-counter caveat directly.** By default each controller replica keeps its OWN counter, so the effective limit is roughly `limit-rps × replicas`. Explain the options: set per-replica limits accounting for replica count, or configure the memcached-backed global limit (`global-rate-limit-*` annotations + a memcached service) for a true cluster-wide limit — and the trade-offs (memcached is a dependency and a single point of coordination).

4. **Produce the annotated Ingress** for the target policy, plus any controller-level ConfigMap settings that must accompany it (forwarded-headers, real-ip CIDRs, default status code).

5. **Design a rollout that won't self-inflict an outage**: start in observation using access logs / metrics to size the limit against real p99 traffic, apply a generous limit first, then tighten. Always whitelist health-check and known internal CIDRs so probes and internal services aren't throttled.

6. **Give the verification and tuning loop**: how to load-test the route (e.g. `hey`/`wrk`), read `nginx_ingress_controller_requests` by status, watch for 503 spikes, and confirm legitimate bursts pass while abuse is shed. Include how to read the controller logs for `limiting requests` entries.

7. **Cover the interactions**: rate limiting does not replace `limit-connections` for slow-loris-style abuse; WAF/ModSecurity is a separate layer; and per-path annotations require separate Ingress objects (annotations are per-Ingress, not per-path).

Flag anything that can cause a self-DoS: a limit set below real traffic, keying on the LB IP (one shared bucket), or forgetting to whitelist health-check probes.

---

Ingress object(s) + routes to protect:
```yaml
[PASTE]
```
Threat / abuse pattern: [DESCRIBE]
Controller replicas + LB/CDN in front: [DESCRIBE]
Real client IP source + trusted CIDRs: [DESCRIBE]
Current annotations / ConfigMap values: [PASTE or "defaults"]

Why this prompt works

ingress-nginx rate limiting looks like a two-line annotation, but the parts that bite in production are invisible in the docs snippet: the counter is per-replica, the limit keys on whatever IP NGINX thinks is the client, and a too-tight limit takes down your own health checks. This prompt forces client-IP correctness first, then sizing, then the shared-state decision.

How to use it

1. Verify real-client-IP detection before touching limit numbers — everything downstream depends on it.
2. Decide per-replica vs. global (memcached) limits based on how many controller replicas you run.
3. Whitelist probes and internal CIDRs up front.
4. Roll out generous, observe, then tighten against measured traffic.

Useful commands

# Inspect the effective annotations on an Ingress
kubectl get ingress <name> -n <ns> -o yaml | grep -A20 annotations

# Controller ConfigMap (real-ip / forwarded headers live here)
kubectl -n ingress-nginx get configmap ingress-nginx-controller -o yaml

# Watch request counts by status code
kubectl -n ingress-nginx get --raw /metrics 2>/dev/null | grep nginx_ingress_controller_requests | grep 'status="503"'

# Controller logs showing throttling
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep -i 'limiting requests'

# Load test a route (confirm bursts pass, abuse is shed)
hey -z 30s -q 50 -c 20 https://api.example.com/expensive

Pattern: per-route rate limit with real-IP and whitelist

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-expensive
  namespace: prod
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "10"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"      # burst = 30
    nginx.ingress.kubernetes.io/limit-connections: "20"
    nginx.ingress.kubernetes.io/limit-req-status-code: "429"     # clients back off
    nginx.ingress.kubernetes.io/limit-whitelist: "10.0.0.0/8,192.168.100.0/24"
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /expensive
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8080

Pattern: controller ConfigMap for real client IP behind an LB

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  use-forwarded-headers: "true"
  compute-full-forwarded-for: "true"
  proxy-real-ip-cidr: "10.0.0.0/8"     # trust only your LB subnet
  # use-proxy-protocol: "true"          # if the LB speaks PROXY protocol instead

Pattern: true cluster-wide limit (memcached-backed)

metadata:
  annotations:
    nginx.ingress.kubernetes.io/global-rate-limit: "100"
    nginx.ingress.kubernetes.io/global-rate-limit-window: "1m"
    nginx.ingress.kubernetes.io/global-rate-limit-key: "$remote_addr"
# Requires the controller configured with a memcached service:
#   global-rate-limit-memcached-host / -port in the controller ConfigMap.

Common findings this catches

• One shared bucket blocks everyone → controller keys on the LB IP; forwarded-headers/real-ip misconfigured.
• Limit is effectively N× intended → per-replica counters and multiple controller replicas; not using the global limit.
• Health checks 503’ing → probe/internal CIDRs missing from limit-whitelist.
• Retry storms after enforcement → 503 default status; switch to 429 so compliant clients back off.
• Per-path policy not applying → annotations set on an Ingress covering multiple paths; split into separate Ingress objects.

When to escalate

• Sustained abuse beyond L7 rate limiting (volumetric/DDoS) — engage the CDN/edge or network team.
• Application-layer attacks (injection, credential stuffing patterns) — add a WAF/ModSecurity layer, out of scope for rate limits.
• Controller-wide instability under load — review controller resources/replica count with the platform team before tightening limits further.