🚨 Kubernetes Error Guide: 'failed to create containerd task' OCI runtime create failed

Exact Error Message

The image pulled fine and scheduling succeeded, but the container runtime cannot start the process. The pod cycles through RunContainerError / CrashLoopBackOff and the event log shows a containerd task creation failure wrapping a runc error:

Events:
  Type     Reason   Age               From     Message
  ----     ------   ----              ----     -------
  Normal   Pulled   12s               kubelet  Successfully pulled image "ghcr.io/acme/api:1.4.0"
  Warning  Failed   11s (x3 over 24s) kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/app/start.sh": permission denied: unknown
  Warning  BackOff  3s  (x3 over 23s) kubelet  Back-off restarting failed container

Other common tails after OCI runtime create failed: runc create failed: include:

... container_linux.go: starting container process caused: exec: "node": executable file not found in $PATH
... unable to start container process: error during container init: error mounting "/host/data" to rootfs: no such file or directory
... process_linux.go: applying cgroup configuration: failed to write 268435456 to memory.max: invalid argument

The wrapper (failed to create containerd task) is generic; the last clause after the final colon is the real cause.

What the Error Means

Starting a container is a layered hand-off: the kubelet calls the CRI (containerd), containerd spawns a shim that calls the OCI runtime (runc), and runc does the low-level work — unpacking the rootfs, setting up namespaces and cgroups, then execing your entrypoint. failed to create containerd task means that chain broke before your process began running.

Because runc fails fast and bubbles its error up through the shim and containerd, the message is a stack of wrappers. Each layer prepends its own context, so you read the message inside-out: ignore the boilerplate prefixes and focus on the final segment, which names the actual syscall or check that failed — a missing executable, a bad mount, a permission denial, or a cgroup write rejection.

This is distinct from image or scheduling errors. The image is already present and the pod has a node. The failure is purely at container creation time, in the runtime layer.

Common Causes

• Entrypoint not executable or missing — exec: "/app/start.sh": permission denied (no +x bit) or executable file not found in $PATH (binary absent / wrong arch).
• Bad bind mount / hostPath — error mounting ... to rootfs: no such file or directory when a hostPath does not exist on the node.
• Cgroup driver mismatch — kubelet and containerd disagree on systemd vs cgroupfs, causing cgroup writes to fail.
• cgroup v2 limit rejection — failed to write ... to memory.max: invalid argument, often an out-of-range or unsupported limit.
• Read-only or full rootfs — the node’s /var/lib/containerd is full or mounted read-only, so the rootfs cannot be created.
• SELinux/AppArmor denial — a security module blocks exec or mount inside the container.
• Architecture mismatch — an amd64 binary on an arm64 node yields exec format error.
• Privileged/device requirements — a container needs a device or capability the runtime cannot grant.

How to Reproduce the Error

Build an image whose entrypoint is not executable:

FROM alpine:3.20
COPY start.sh /app/start.sh
# note: no chmod +x
ENTRYPOINT ["/app/start.sh"]

kubectl run runc-fail --image=ghcr.io/acme/badentry:1.0
kubectl get pod runc-fail

NAME        READY   STATUS                 RESTARTS   AGE
runc-fail   0/1     RunContainerError      0          9s

kubectl describe pod runc-fail shows OCI runtime create failed: ... permission denied.

Diagnostic Commands

# Read the full wrapped runtime error
kubectl describe pod <POD> | grep -A8 Events

# On the node, inspect what containerd/runc actually attempted
sudo crictl ps -a | grep <POD>
sudo crictl inspect <CONTAINER_ID> | jq '.status.reason, .info.runtimeSpec.process.args'
sudo journalctl -u containerd --no-pager --since "10 min ago" | grep -i 'oci\|runc\|shim'

# Confirm kubelet and containerd agree on the cgroup driver
sudo crictl info | jq '.config.systemdCgroup'
sudo cat /var/lib/kubelet/config.yaml | grep -i cgroupDriver

# Check node disk for the containerd state dir and cgroup version
df -h /var/lib/containerd
stat -fc %T /sys/fs/cgroup/   # cgroup2fs == v2

# Verify the image's entrypoint and architecture without running it
crane config ghcr.io/acme/api:1.4.0 | jq '.config.Entrypoint, .architecture'

crictl inspect plus the journalctl -u containerd lines reveal the exact runc invocation and the syscall that failed.

Step-by-Step Resolution

1. Isolate the final clause. Strip the wrappers and read only the text after the last colon — that names the real failure.

2. Fix an entrypoint problem. For permission denied, mark the script executable in the image; for executable file not found, fix the path or install the binary:

COPY start.sh /app/start.sh
RUN chmod +x /app/start.sh
ENTRYPOINT ["/app/start.sh"]

3. Fix a mount/rootfs error. Ensure any hostPath exists on the node (or set type: DirectoryOrCreate), and confirm the path is correct. no such file or directory at rootfs mount time is almost always a bad volume source.

4. Reconcile the cgroup driver. Both kubelet and containerd should use systemd on systemd hosts. Set SystemdCgroup = true in /etc/containerd/config.toml and cgroupDriver: systemd in the kubelet config, then restart both.

5. Resolve cgroup v2 limit rejections. invalid argument writing memory.max usually means an unsupported or out-of-range limit. Set sane integer limits and confirm the node runs cgroup v2.

6. Free or remount the rootfs store. If /var/lib/containerd is full or read-only, reclaim space (crictl rmi --prune) or fix the mount, then restart containerd.

7. Address arch / security denials. For exec format error, schedule onto the matching architecture or build multi-arch. For SELinux/AppArmor denials, adjust the profile or seLinuxOptions rather than disabling enforcement wholesale.

Prevention and Best Practices

• Bake an entrypoint smoke test into CI: run the image locally before publishing so a non-executable or missing binary fails the build, not the cluster.
• Keep kubelet and containerd on the same cgroup driver across the whole fleet; a mismatch is a recurring source of task-creation failures.
• Prefer named volumes and PVCs over hostPath; when hostPath is unavoidable, use DirectoryOrCreate and validate the node path.
• Monitor node disk for /var/lib/containerd and /var; a full runtime store breaks every new container at once.
• Build and publish multi-arch images for mixed-architecture clusters to eliminate exec format error.
• Standardize on cgroup v2 across nodes and test resource limits there before rolling out.

Related Errors

• CreateContainerError — the broader container-creation failure family.
• CreateContainerConfigError — config (secret/configmap) problems before runtime start.
• CrashLoopBackOff — the back-off state these failures land in.
• OOMKilled — a container that starts but is killed for exceeding memory.

Frequently Asked Questions

Why is the error message so long and nested? Each layer in the start chain — containerd, the shim, runc, and Linux container init — wraps the underlying error with its own context. Read it inside-out; only the final clause after the last colon describes the actual fault.

The image runs fine with docker run locally but fails in the cluster. Why? The usual culprits are an architecture mismatch (local amd64 vs an arm64 node), a hostPath that exists on your laptop but not the node, or a cgroup-driver difference. Inspect with crictl on the node to see the real runtime spec.

What does failed to write ... memory.max: invalid argument mean? On cgroup v2, runc writes your memory limit to memory.max. An out-of-range, unsupported, or malformed value is rejected by the kernel. Set a valid integer limit and confirm the node’s cgroup version.

Is this an image problem or a node problem? Both are possible. Entrypoint/permission/arch causes are baked into the image; cgroup-driver, disk-full, hostPath, and SELinux causes are node-side. The final clause of the error tells you which side to fix.

How do I tell this apart from CreateContainerConfigError? CreateContainerConfigError fails while assembling the container config (a missing secret or configmap key) before the runtime is invoked. failed to create containerd task fails inside the runtime as it tries to start the process — the config was already valid.