🚨 Docker Error Guide: 'OCI runtime exec failed: read-only file system' — Add Writable Paths

Overview

This error appears when a process inside a container tries to write to the container filesystem, but the container was started in read-only mode:

OCI runtime exec failed: exec failed: unable to start container process: open /proc/self/task/1/attr/exec: read-only file system: unknown

The read-only file system here is EROFS. The container was launched with --read-only (or read_only: true in Compose), which mounts the container’s root filesystem read-only for security hardening. Something the process or runtime needs to write — a lock file, a PID file, a cache, a runtime state path — lives on that now-read-only root. The fix is not to abandon read-only mode, but to provide writable space (tmpfs or a volume) for exactly the paths that need it.

Symptoms

• Containers fail to start, or docker exec fails, with read-only file system: unknown.
• The same image runs fine without --read-only and fails the moment read-only mode is added.
• Application logs show EROFS / “Read-only file system” when opening files under /tmp, /var/run, /run, /var/cache, or an app-specific data directory.
• Web servers (nginx), databases, and apps that write PID files, sockets, or temp files fail during startup under read-only.
• Works in dev (no hardening) but breaks in a hardened production/Kubernetes securityContext.

Common Root Causes

• --read-only with no writable mounts — the root is read-only and the process has nowhere to write its temp/lock/socket files.
• Applications that write to /tmp for scratch data, sessions, or compilation, which is on the read-only root by default.
• Runtime state paths like /var/run or /run where daemons place PID files and unix sockets.
• Caches written under /var/cache, ~/.cache, or a framework-specific directory.
• A process expecting to write its own config or log files into an application directory baked into the image.

Diagnostic Workflow

First confirm the container really is read-only and find out exactly which path failed — the path in the error message is the clue:

docker inspect <container> --format '{{.HostConfig.ReadonlyRootfs}}'
docker inspect <container> --format '{{json .Mounts}}' | jq
journalctl -u docker --since '10 min ago' | grep -i 'read-only\|erofs'

Reproduce interactively to see every path the process tries to write. Run read-only and watch it fail, then run writable to compare:

docker run --rm -it --read-only myimage:latest sh -c 'touch /tmp/x; echo ok'
# EROFS on /tmp confirms /tmp needs to be writable

Trace filesystem writes if the failing path is not obvious from the error, using strace inside a writable run:

docker run --rm -it myimage:latest sh -c 'apk add strace; strace -f -e trace=openat,mkdir myapp 2>&1 | grep -i "O_WRONLY\|O_CREAT"'

Once you know the paths, add a tmpfs for each so the root stays read-only but those directories are writable in RAM:

docker run --read-only \
  --tmpfs /tmp \
  --tmpfs /run \
  myimage:latest

In Compose, keep read_only: true and declare tmpfs (and any persistent volume) explicitly:

services:
  web:
    image: nginx:1.27
    read_only: true
    tmpfs:
      - /tmp
      - /var/run
      - /var/cache/nginx
    volumes:
      - app-data:/var/lib/app     # persistent, writable
volumes:
  app-data:

Example Root Cause Analysis

A platform team hardened all workloads with read_only: true and their nginx sidecar immediately crash-looped with read-only file system.

docker inspect confirmed ReadonlyRootfs: true. Running the image interactively read-only and tailing the error showed nginx failing to create its temp paths: it writes to /var/cache/nginx (proxy/fastcgi temp files) and a PID file under /var/run. Both live on the read-only root, so EROFS killed startup. The image was fine; read-only mode simply removed the writable scratch space nginx assumes it has.

The fix kept the hardening and added tmpfs mounts for precisely those paths:

services:
  web:
    image: nginx:1.27
    read_only: true
    tmpfs:
      - /var/cache/nginx
      - /var/run
      - /tmp

nginx started cleanly, and because the temp paths are tmpfs (RAM-backed and ephemeral), the security benefit of an immutable root filesystem was preserved. The root cause was not read-only mode being wrong — it was that nginx’s known writable directories had not been provisioned as writable mounts.

Prevention Best Practices

• When enabling --read-only, enumerate the writable paths each image needs (/tmp, /run, caches) and provide --tmpfs for ephemeral ones up front.
• Use named volumes or bind mounts for data that must persist; use tmpfs for scratch that should vanish with the container.
• Test images under --read-only in CI so missing writable paths are caught before production hardening.
• Prefer images designed for read-only roots (many official images document their writable paths) and configure apps to write temp data to a mounted location.
• Document each service’s required writable mounts alongside its read_only: true setting so the pairing is never separated.

Quick Command Reference

# Is the container read-only, and what is mounted?
docker inspect <c> --format '{{.HostConfig.ReadonlyRootfs}}'
docker inspect <c> --format '{{json .Mounts}}' | jq

# Reproduce and find the failing write path
docker run --rm -it --read-only myimage sh -c 'touch /tmp/x'

# Run read-only with writable tmpfs for scratch paths
docker run --read-only --tmpfs /tmp --tmpfs /run myimage

# Persistent writable data with an otherwise read-only root
docker run --read-only --tmpfs /tmp -v app-data:/var/lib/app myimage

Conclusion

OCI runtime exec failed: ... read-only file system is EROFS from a container running with --read-only: the process needs to write somewhere on a root you have made immutable. The fix is targeted, not a rollback — inspect the failing path from the error, reproduce to enumerate every writable path the process needs, then mount --tmpfs for ephemeral scratch and a volume for anything that must persist. Done this way you keep the security win of an immutable root filesystem while giving each application exactly the writable space it requires.