Production Incident Simulation
It’s an incident: a multi-service app is throwing errors for some users. Multiple faults are planted at once — a DB connection failure, an unhealthy dependency, a full volume, a wrong env var, a reverse-proxy routing bug, and a resource-starved container. Diagnose systematically, repair, validate every service is healthy, and produce a brief incident writeup.
Learning objectives
- Establish what is and isn’t working
- Diagnose each independent fault
- Apply repairs in a safe order
- Validate all services report healthy
- Write a concise incident summary
Before you start
Required software: Docker Engine 24+ with Compose v2
Prerequisites: Docker Monitoring & Troubleshooting path, Docker CI/CD & Production path
- Download and unzip the lab.
- Start the stack:
docker compose up -d. - Load the app — parts of it work, parts fail. Begin triage.
Instructions
Full instructions are part of Individual Pro
Unlock the step-by-step instructions, the solution guide, validation scripts, and downloadable files.
Hints
💡 Where do I start?
Map the topology first (proxy → web → api → db + workers). Check health/logs per hop from the edge inward; treat each fault independently so one red herring doesn’t hide another.
💡 Some services flap under load
Look for containers hitting memory/CPU limits (docker stats) and a volume at 100% — a full volume can make a “healthy” service start failing writes.
Validate your work
From the lab folder, run:
bash validation/validate.shThe script reads only your local Docker state, never transmits data off your machine, and returns a status code:
0— Lab successfully completed1— Validation failed2— Required dependency missing3— Lab environment not running4— Configuration error5— Validation timed out
Note: validation targets Linux first. On Docker Desktop (macOS/Windows) some host-level checks (e.g. disk usage) may report differently; each script documents its limitations.
Solution
The full solution + troubleshooting explanation is available to Individual Pro members. Try to solve it first — that’s where the learning is.
Reset the lab
docker compose down -v && docker compose up -dfrom the original lab files restores the starting (broken) state.
Interview questions
- How do you triage a partial outage across multiple services?
- What signals tell you a container is resource-starved?
- What belongs in a good incident summary?
Finished this lab?
Sign in to record your completion and track progress across the path. Completion is recorded server-side.