Linux Kernel Keyring (keyctl) Inspection & Debug Prompt
Inspect and reason about the kernel keyrings (user, session, process, thread, persistent) behind Kerberos tickets, fscrypt keys, NFS/CIFS creds, and dm-crypt — and debug why a key is missing, expired, or not visible to a service.
- Target user
- Linux admins debugging Kerberos, fscrypt, or kernel-keyring-backed auth and encryption
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior Linux engineer who understands the kernel key-management facility — the keyring hierarchy, key types, permissions, expiry, and how keyrings are (and aren't) inherited across processes and sessions.
I will provide:
- The symptom: "key was rejected by service", a service that can't find an fscrypt/Kerberos/CIFS key, or a key that vanishes unexpectedly
- `keyctl show @u` / `keyctl show @s` / `keyctl show @p` output for the affected user and the service
- The key type involved (logon, user, keyring, fscrypt-provisioning, big_key, ceph, etc.) if known
- How the service runs (interactive login, systemd unit, cron, container) — this determines which keyrings it sees
- For Kerberos: `klist` output and the cred cache type (KEYRING: vs FILE:)
Your job:
1. **Map the keyring hierarchy for THIS case** — explain thread → process → session → user → user-session → persistent keyrings, and which one the service's keys actually live in vs which one it's looking in.
2. **Diagnose the visibility gap** — the most common failure: keys added to an interactive session keyring aren't visible to a systemd service or cron job that runs in a *different* session. Identify whether this is the problem here and why.
3. **Read permissions & expiry** — decode the key permission mask (possessor/user/group/other × view/read/write/search/link/setattr) and any `timeout`/expiry so I can see if the service even has search rights, or if the key has simply expired.
4. **Pick the right keyring** — recommend the correct target keyring for the use case (persistent keyring via `keyctl get_persistent` for services that must survive session changes; user keyring for per-user; explicit linking when a service must reach a key).
5. **Fix and harden** — give the exact `keyctl` commands (add/link/setperm/timeout) or the config change (e.g. Kerberos `default_ccache_name = KEYRING:persistent:%{uid}`, fscrypt key provisioning) and how to confirm the service now resolves the key.
Output as: (a) a plain-language map of where the key is vs where the service is looking, (b) the root cause (wrong keyring, missing search permission, or expiry), (c) the exact remediation commands/config, (d) the verification step proving the service resolves the key.
Verify before acting: do not blanket-grant broad key permissions or move secrets to a world-searchable keyring to "make it work" — fix the targeting (correct keyring + minimal search rights) and verify in a fresh session matching how the service actually runs.
Why this prompt works
The Linux kernel keyring is invisible plumbing that underpins a surprising amount of production: Kerberos credential caches, fscrypt encryption keys, NFS/CIFS authentication, and dm-crypt all lean on it. When something keyring-backed breaks, the error (“Key was rejected by service”, “Required key not available”) is famously unhelpful, and the root cause is almost always the keyring hierarchy — which keyring a key landed in versus which keyring the failing process searches. This prompt makes the AI map that hierarchy for your specific case instead of leaving you to memorize the thread/process/session/user/persistent ladder.
The highest-yield insight it encodes is the session-visibility gap: a key you add by hand in an SSH shell lives in your interactive session keyring, and a systemd service or cron job runs in a completely different session, so it never sees the key. People burn hours on this because their manual test “works” while the service keeps failing. By telling the model to check whether the service’s execution context matches where the key was placed, the prompt catches the single most common keyring bug immediately.
It also forces the model to read the often-ignored details — the permission mask and the expiry timeout — so “the service can’t find the key” gets correctly reclassified as “the service lacks search permission” or “the key timed out.” The fix then targets the right keyring (often the persistent keyring for long-lived services) with minimal permissions rather than the lazy and dangerous move of widening a key to world-searchable. The AI diagnoses the targeting and drafts the keyctl or config fix; you verify it in a context that matches how the service truly runs.