🦭 Podman Error: 'crun: setrlimit RLIMIT_NOFILE' When Starting a Container

Exact Error Message

Error: OCI runtime error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted

Under a systemd service or Quadlet unit the same failure usually surfaces wrapped in the unit’s log:

podman[4127]: Error: crun: setrlimit `RLIMIT_NOFILE`: Operation not permitted
systemd[1]: web.service: Main process exited, code=exited, status=126/n/a
systemd[1]: web.service: Failed with result 'exit-code'.

What It Means

Before it execs your container’s entrypoint, the OCI runtime — crun on most modern installs, runc elsewhere — applies the resource limits from the container’s OCI spec by calling setrlimit(). RLIMIT_NOFILE is the maximum number of open file descriptors. The kernel allows an unprivileged process to raise its soft limit up to its own hard limit, and to lower the hard limit, but never to raise the hard limit above what it inherited. Attempting that returns EPERM, which crun reports verbatim as Operation not permitted.

So this error means the nofile limit the container asked for is larger than the hard limit of the process that launched Podman. That request can come from three places: an explicit --ulimit nofile=... flag, a default_ulimits entry in containers.conf, or a value baked into an image or Kubernetes YAML. Rootless Podman hits it far more than rootful, because a rootless container runs entirely as your user with your inherited limits — there is no privileged step that can raise the ceiling. A rootful podman run by root usually starts with a very high hard limit and rarely trips this, which is why the same command can work under sudo and fail as a normal user. The fix is either to lower what the container asks for, or to raise the hard limit of the launching context.

Common Causes

• A rootless user session inherits a low hard limit (for example 4096) from PAM limits or the systemd user manager.
• containers.conf sets default_ulimits = ["nofile=1048576:1048576"], above the user’s hard limit.
• An explicit --ulimit nofile=65535:65535 on the command line exceeds ulimit -Hn.
• A systemd service or Quadlet unit runs Podman without a LimitNOFILE= high enough for the container’s request.
• DefaultLimitNOFILE in /etc/systemd/user.conf is lower than the value in /etc/systemd/system.conf, so user services get a tighter ceiling than system services.
• Kubernetes YAML played with podman kube play carries a nofile request tuned for a cluster with much higher node limits.

Diagnostic Commands

Check the limits of the shell that is launching Podman. The hard limit is the ceiling that matters:

ulimit -Sn
ulimit -Hn

Read the limits of a live process directly from procfs, which is authoritative for services where your shell is not representative:

grep 'Max open files' /proc/self/limits
systemctl --user show web.service -p LimitNOFILE -p LimitNOFILESoft

Find out what Podman is asking for by inspecting the merged configuration:

podman info --format '{{.Host.OCIRuntime.Name}} {{.Host.OCIRuntime.Version}}'
grep -rn 'default_ulimits' /usr/share/containers/containers.conf /etc/containers/containers.conf ~/.config/containers/containers.conf 2>/dev/null

Reproduce with an explicit value to confirm the boundary — find the highest number that starts successfully:

podman run --rm --ulimit nofile=4096:4096 docker.io/library/alpine:latest sh -c 'ulimit -n'
podman run --rm --ulimit nofile=65535:65535 docker.io/library/alpine:latest sh -c 'ulimit -n'

Check the systemd-wide defaults that seed every session and service:

grep -E '^\s*DefaultLimitNOFILE' /etc/systemd/system.conf /etc/systemd/user.conf
systemctl show -p DefaultLimitNOFILE

Step-by-Step Resolution

1. Establish the ceiling. If ulimit -Hn prints 4096 and your container asks for 65535, you have found the conflict and only need to decide which side to change:

ulimit -Hn

2. For a one-off run, ask for a value at or below the hard limit. Note the SOFT:HARD ordering — the soft value comes first:

podman run --rm --ulimit nofile=4096:4096 docker.io/library/nginx:alpine

3. If a config file is imposing the oversized default, correct it in the rootless override rather than editing the vendored file under /usr/share/containers:

# ~/.config/containers/containers.conf
[containers]
default_ulimits = [
  "nofile=4096:4096",
]

4. To genuinely raise the ceiling for interactive logins, add a limits drop-in for the user and start a fresh login session so PAM applies it:

sudo tee /etc/security/limits.d/90-podman-nofile.conf <<'EOF'
jane  soft  nofile  65535
jane  hard  nofile  1048576
EOF

5. For systemd-managed workloads, PAM limits do not apply — set the limit on the unit itself. This is the correct fix for a Quadlet-generated service:

# ~/.config/containers/systemd/web.container
[Container]
Image=docker.io/library/nginx:alpine
PublishPort=8080:80

[Service]
LimitNOFILE=65535

6. Reload and restart, then verify the container sees the limit you intended rather than the one it asked for:

systemctl --user daemon-reload
systemctl --user restart web.service
podman exec web sh -c 'ulimit -Sn; ulimit -Hn'

If the limit still will not take for user services, raise the systemd-wide user default, which caps every unit the user manager starts regardless of what individual units request:

sudo mkdir -p /etc/systemd/user.conf.d
sudo tee /etc/systemd/user.conf.d/nofile.conf <<'EOF'
[Manager]
DefaultLimitNOFILE=65535:1048576
EOF
sudo systemctl daemon-reexec

A matching drop-in under /etc/systemd/system.conf.d/ does the same for system services. Both require a re-exec of systemd and a fresh user session to take effect, which is why “I changed it and nothing happened” is the usual follow-up complaint. Quadlet units in particular need systemctl --user daemon-reload after any edit; see Podman error: Quadlet unit failed to start for the wider set of unit-generation failure modes.

Prevention

• Set default_ulimits in containers.conf to a value your lowest-privileged host can actually satisfy, not the largest number you have seen recommended.
• Ship LimitNOFILE= explicitly in Quadlet and systemd units instead of relying on inherited session limits.
• Keep DefaultLimitNOFILE consistent between /etc/systemd/system.conf and /etc/systemd/user.conf so rootful and rootless behave the same.
• Include ulimit -Hn in host provisioning checks and fail the build when it is below your platform’s documented floor.
• Avoid copying nofile values out of Kubernetes manifests onto single-host Podman without re-checking the node limits they assumed.
• Test new images as the target rootless user, not under sudo, so limit conflicts appear before production.

Related Errors

• container exited with code 137 — the container was killed by the OOM killer or SIGKILL, a memory limit rather than a descriptor limit; see Podman error: exited with code 137.
• cannot bind to port 80: permission denied — a rootless capability limit, not an rlimit; see Podman error: rootless bind to privileged port.
• too many open files from inside the application — the limit applied successfully but is too low for the workload.
• OCI runtime error: crun: cannot set memory limit — the same class of spec-application failure against a different cgroup or rlimit control.

Frequently Asked Questions

Why does it work with sudo podman but not rootless? Root starts with a very high hard limit and holds CAP_SYS_RESOURCE, so it can raise limits freely. A rootless container inherits your user’s hard limit with no way to exceed it, so the same request that root satisfies returns EPERM for you.

What is the correct order in --ulimit nofile=A:B? A is the soft limit and B is the hard limit, and B must not exceed the launching process’s hard limit. Setting only --ulimit nofile=65535 applies that value to both, which is a common way to trip the ceiling unintentionally.

Why did editing /etc/security/limits.d/ change nothing for my service? PAM limits apply to login sessions, not to units started by systemd. Services take their limits from LimitNOFILE= on the unit or from DefaultLimitNOFILE in the manager config. Set it on the unit for predictable results.

Should I just set nofile to 1048576 everywhere? Only if the hosts can support it and the workload needs it. Very high descriptor limits raise memory use for processes that size internal tables from the limit, and they mask descriptor leaks that a sane ceiling would surface early. Pick a documented value per workload class. For more container-runtime fixes, see the Podman guides.