🚨 Linux Error Guide: 'cgroup: fork rejected by pids controller' — Fix pids.max Limits

Overview

The kernel’s pids cgroup controller caps how many processes (tasks) a cgroup may hold, and logs this to dmesg when a fork() is blocked because the cgroup is at its ceiling:

cgroup: fork rejected by pids controller in /system.slice/myapp.service

Inside the affected process, the blocked fork()/clone() surfaces as EAGAIN:

fork: retry: Resource temporarily unavailable
bash: fork: Cannot allocate memory

This is distinct from a system-wide pid_max or memory exhaustion: it’s a per-cgroup limit. systemd sets pids.max from a unit’s TasksMax (and a default DefaultTasksMax), so a service can hit its own task ceiling while the rest of the machine has plenty of PIDs and RAM to spare — or a runaway process/thread leak can legitimately be trying to blow past the limit.

Symptoms

• dmesg shows fork rejected by pids controller in /<cgroup path>.
• A specific service can’t spawn workers/threads while the host is otherwise healthy.
• Applications log fork: Resource temporarily unavailable or Cannot allocate memory despite free RAM.
• systemctl status <unit> shows Tasks: N (limit: N) with current at the limit.
• The failure is scoped to one service or container, not the whole system.

Common Root Causes

• TasksMax too low for the workload. The unit legitimately needs more tasks than DefaultTasksMax (or an explicit TasksMax) allows.
• A thread or process leak. The app spawns threads/children without reaping them, climbing toward the limit — the limit is doing its job.
• A fork bomb / runaway. Buggy code or a bad deploy multiplies processes rapidly and slams into pids.max.
• Container runtime pids limit. --pids-limit (Docker) or a Kubernetes pod pids limit sets a low pids.max on the container’s cgroup.
• Default too low for many-worker apps. Databases, CI runners, and app servers with large worker/thread pools exceed a conservative default.
• Zombie accumulation. Unreaped children count against the cgroup until harvested.

Diagnostic Workflow

Identify which cgroup hit the wall and how close to the limit it is right now:

dmesg -T | grep -i 'fork rejected by pids'
systemctl status myapp.service | grep -i tasks     # "Tasks: N (limit: M)"

Read the controller’s own accounting directly (cgroup v2 path from the dmesg line):

cat /sys/fs/cgroup/system.slice/myapp.service/pids.current
cat /sys/fs/cgroup/system.slice/myapp.service/pids.max
cat /sys/fs/cgroup/system.slice/myapp.service/pids.events   # 'max' counter = times throttled

Decide leak vs. legitimate need — is the task count climbing without bound?

systemd-cgtop -1                       # per-cgroup task counts, live
ps -o pid,nlwp,comm --ppid $(systemctl show -p MainPID --value myapp.service)
ps --ppid <pid> -L | wc -l             # threads under the main process

Check the current and default ceilings:

systemctl show myapp.service -p TasksMax
systemctl show -p DefaultTasksMax

Example Root Cause Analysis

After migrating a CI runner into its own systemd service, jobs began failing intermittently with fork: Resource temporarily unavailable, and dmesg showed cgroup: fork rejected by pids controller in /system.slice/ci-runner.service.

systemctl status ci-runner displayed Tasks: 512 (limit: 512) — the unit had inherited a DefaultTasksMax of 512. Watching systemd-cgtop during a build showed task count rising smoothly to exactly 512 and stalling there, not spiking chaotically. Cross-checking with pids.events showed the max throttle counter incrementing in lockstep with the failures. This pattern — a steady climb to a round limit, not an explosive spike — pointed to a legitimately parallel workload, not a fork bomb: the runner spawned many short-lived compiler and test processes concurrently that briefly exceeded 512 tasks.

The fix was to raise TasksMax for that unit via a drop-in (TasksMax=8192) and reload, after confirming the host’s global pid_max and memory had ample headroom. Had the count instead spiked unbounded, the correct response would have been the opposite: leave the limit as the safety net and fix the leaking process, because pids.max had prevented a fork bomb from taking down the whole host.

Prevention Best Practices

• Set TasksMax per unit to match the real concurrency of many-worker services (databases, CI, app servers) instead of relying on the conservative default.
• Keep DefaultTasksMax as a fleet-wide safety net — it stops a single fork bomb from exhausting the host — and raise it only where justified.
• Monitor pids.current vs pids.max (via systemd-cgtop or cgroup metrics) and alert before services hit the ceiling.
• Treat an unbounded climb as a bug: fix thread/child leaks and reap zombies rather than endlessly raising the limit.
• For containers, size --pids-limit / pod pids limits to the workload; too low breaks apps, too high removes the fork-bomb guard.

Quick Command Reference

# Which cgroup, and how close to the limit
dmesg -T | grep -i 'fork rejected by pids'
systemctl status <unit> | grep -i tasks
cat /sys/fs/cgroup/system.slice/<unit>/pids.{current,max,events}

# Live per-cgroup task counts
systemd-cgtop

# Raise a unit's task limit via a drop-in
sudo systemctl edit <unit>       # add: [Service]\nTasksMax=8192
sudo systemctl daemon-reload && sudo systemctl restart <unit>

# Check global headroom before raising limits
cat /proc/sys/kernel/pid_max
systemctl show -p DefaultTasksMax

Conclusion

cgroup: fork rejected by pids controller is a per-cgroup task-count ceiling (pids.max, set from systemd’s TasksMax) being hit — not a system-wide PID or memory shortage. Read the dmesg line to find the cgroup, compare pids.current to pids.max, and watch whether the count climbs steadily (a legitimately parallel workload that needs a higher TasksMax) or spikes without bound (a leak or fork bomb the limit is correctly stopping). Raise the unit’s limit only when the workload justifies it, keep a sane default as a safety net, and fix leaks rather than removing the guardrail.