Linux PAM Authentication Debugging Prompt

You are a senior Linux sysadmin with deep PAM experience — local accounts, SSSD/LDAP/AD integration, MFA modules, password quality enforcement. You can read a PAM stack top-to-bottom and tell exactly where it fails.

I will provide:
- The symptom (login fails for valid creds, account locked, "permission denied" with no useful error, MFA prompt doesn't appear, sudo fails)
- The specific PAM service involved (`sshd`, `login`, `sudo`, `gdm`, `su`, etc.) — `/etc/pam.d/<service>`
- Relevant /etc/pam.d include files (e.g., `common-auth`, `password-auth`, `system-auth`)
- Output of `/var/log/auth.log` (Debian/Ubuntu) or `/var/log/secure` (RHEL) around the failure
- `getent passwd <user>` and `getent shadow <user>` (where permitted)
- For directory: `sssctl user-checks <user>` or `getent passwd <user> -s sss`
- Faillock state: `faillock --user <user>`

Your job:

1. **Walk the PAM stack** for the specific service:
   - PAM files use modules (`pam_unix.so`, `pam_sss.so`, etc.) in 4 phases: `auth`, `account`, `password`, `session`
   - Each phase runs modules top-to-bottom with control flags: `required`, `requisite`, `sufficient`, `optional`, `[<actions>]`
   - **`required`** → must succeed; failure registered but continues
   - **`requisite`** → must succeed; failure causes IMMEDIATE return
   - **`sufficient`** → success short-circuits the stack with success; failure ignored
   - **`optional`** → result ignored unless it's the only module in the stack
2. **Identify the failing phase**:
   - **`auth` failure** → bad password, account locked, MFA fail
   - **`account` failure** → account expired, password expired, login time restriction, group restriction
   - **`password` failure** → only during password change; quality / history checks failed
   - **`session` failure** → environment setup, sudoers issue, pam_systemd
3. **Decode log lines**:
   - `pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 ... user=james` → wrong password OR account in `nullok` / shadow issue
   - `pam_faillock(sshd:auth): User account locked due to <N> failed logins` → faillock triggered
   - `pam_sss(sshd:auth): authentication failure ... user=james` → SSSD didn't authenticate; check SSSD logs
   - `pam_unix(sudo:account): account james has expired (account expired)` → `chage` shows expiry; reset with `chage -E -1`
   - `pam_systemd(login:session): Failed to create session: Activation of org.freedesktop.login1 timed out` → systemd-logind issue
4. **Common scenarios**:
   - **`faillock` lockout** — `faillock --user <user>` shows attempts; `faillock --user <user> --reset` clears. Lockout policy in `/etc/security/faillock.conf`.
   - **Password expired** — `chage -l <user>` shows; `passwd <user>` resets.
   - **`access.conf` denial** — `pam_access.so` reads `/etc/security/access.conf` for per-user/group/host rules.
   - **`pwquality` rejection during password change** — minimum length, character classes; tune `/etc/security/pwquality.conf`.
   - **SSSD not finding user** — directory join broken; `sssctl domain-status <domain>`, `realm list`, `kinit` for AD.
   - **MFA module not prompting** — module ordering; MFA must come before sufficient pam_unix.
   - **Sudo asks for password despite NOPASSWD** — wrong sudoers file, syntax error in `/etc/sudoers.d/`, or `requiretty`.
5. **For PAM debug mode**:
   - Add `debug` to a module line for verbose logging: `auth sufficient pam_unix.so debug`
   - Restart the service or just retry login
   - `journalctl -t sshd` or `journalctl -u sshd --since "1 hour ago"`
6. **For SSSD-specific**:
   - `sssctl logs-fetch /tmp/sssd-logs.tar.gz` packages logs
   - `sss_cache -E` flushes cache (try after directory changes)
   - `realm leave && realm join` if join is corrupted (DESTRUCTIVE; reauth required)

Mark DESTRUCTIVE: changing PAM config without an open root shell (lockout risk), `realm leave`, `pam_unix.so nullok` (allows empty passwords).

---

Symptom: [DESCRIBE]
Affected service: [sshd / login / sudo / etc.]
`/etc/pam.d/<service>`:
```
[PASTE]
```
Included files (`common-auth`, `password-auth`, etc.):
```
[PASTE]
```
Auth log excerpt:
```
[PASTE]
```
`getent passwd <user>` and `faillock --user <user>`:
```
[PASTE]
```
Directory integration (if relevant): SSSD / nslcd / Kerberos config sanitized:
```
[PASTE]
```

Why this prompt works

PAM failures present as “authentication failed” with no hint at which module rejected. The stack control flow (required vs requisite vs sufficient) is subtle. This prompt forces a phase-by-phase walk with the actual log evidence.

How to use it

1. Capture /var/log/auth.log or /var/log/secure around the failure timestamp.
2. Always include the relevant /etc/pam.d/ file AND its includes (common-auth, system-auth).
3. Check faillock state for any “locked out” complaint before guessing.
4. Keep a root shell open while editing PAM. Many fixes go wrong on the first try.

Useful commands

# PAM config
cat /etc/pam.d/sshd
cat /etc/pam.d/common-auth /etc/pam.d/common-account     # Debian/Ubuntu
cat /etc/pam.d/system-auth /etc/pam.d/password-auth      # RHEL

# Logs
sudo journalctl -t sshd --since "1 hour ago"
sudo journalctl -u sshd --since "1 hour ago"
sudo tail -200 /var/log/auth.log         # Debian/Ubuntu
sudo tail -200 /var/log/secure           # RHEL

# Faillock state
sudo faillock --user <user>
sudo faillock --user <user> --reset
cat /etc/security/faillock.conf

# Tally state (older, pam_tally2)
sudo pam_tally2 --user <user>
sudo pam_tally2 --user <user> --reset

# Account state
chage -l <user>
sudo passwd -S <user>                    # locked/active status
sudo usermod -U <user>                   # unlock
sudo usermod -e -1 <user>                # remove expiry (-1 = never)
sudo chage -E -1 <user>                  # remove account expiration

# User lookup
getent passwd <user>
getent shadow <user>                     # root only
id <user>
groups <user>

# Directory
sudo realm list
sudo sssctl domain-status <domain>
sudo sssctl user-checks <user>
sudo sss_cache -E                        # flush cache (heavy)
sudo systemctl status sssd
sudo journalctl -u sssd --since "1 hour ago"

# Test PAM stack
sudo pamtester sshd <user> authenticate

# Debug a specific module (add 'debug' temporarily)
# In /etc/pam.d/sshd:
# auth     required     pam_unix.so debug
# Then retry login and check logs

# pwquality
cat /etc/security/pwquality.conf
echo "newpassword" | pwscore               # check quality score

# Access control (pam_access)
cat /etc/security/access.conf

# sudo
sudo -l -U <user>                          # effective sudo rules
sudo visudo -c                             # syntax check

Decision tree

"Login fails"
│
├── auth.log says "authentication failure" → wrong password OR locked
│   ├── faillock --user <user> shows count → reset OR wait
│   └── No lockout → password issue (incl. expired)
│
├── auth.log says "account expired" → chage -l, fix expiry
│
├── auth.log says "User account is locked" → usermod -U
│
├── auth.log says "Permission denied (publickey,...)" → SSH key issue, not PAM
│
├── auth.log says "pam_systemd ... failed" → logind issue
│
└── No useful log → add `debug` to suspect module, retry, re-read log

Common findings this catches

• pam_faillock triggered after N failures → reset with faillock --user <user> --reset; investigate source of bad attempts (compromised key, brute force).
• Account password expired silently → chage -l <user>; reset with passwd <user>.
• pam_access.so denying via group rule → /etc/security/access.conf; modify or move user.
• SSSD cache stale after AD group change → sss_cache -E then retry.
• pam_unix.so before pam_sss.so with sufficient → local users authenticate, AD users fail because the stack short-circuits.
• MFA module after pam_unix.so sufficient → never reaches MFA. Reorder.
• pwquality rejection on otherwise reasonable password → tighten /etc/security/pwquality.conf only if policy requires.
• sudo asks for password despite NOPASSWD → sudoers syntax error elsewhere in /etc/sudoers.d/ invalidates the file.

Common PAM stack (sshd, RHEL system-auth)

# /etc/pam.d/sshd (uses system-auth/password-auth)

auth     required     pam_faillock.so preauth silent deny=5 unlock_time=900
auth     [success=1 default=bad] pam_unix.so
auth     [default=die] pam_faillock.so authfail deny=5 unlock_time=900
auth     sufficient   pam_faillock.so authsucc deny=5 unlock_time=900
auth     required     pam_sss.so use_first_pass
auth     required     pam_deny.so

account  required     pam_unix.so
account  sufficient   pam_localuser.so
account  sufficient   pam_succeed_if.so uid < 1000 quiet
account  required     pam_sss.so

password required     pam_pwquality.so retry=3
password sufficient   pam_unix.so sha512 shadow nullok use_authtok
password required     pam_sss.so use_authtok
password required     pam_deny.so

session  optional     pam_keyinit.so revoke
session  required     pam_limits.so
session  required     pam_unix.so
session  optional     pam_sss.so
session  optional     pam_systemd.so

When to escalate

• Authentication failures in lockstep across many users — directory or kerberos issue; engage identity team.
• Repeated lockouts suggesting attack — coordinate with security; correlate with last, lastb.
• pam_systemd / logind not creating user sessions — likely systemd-level issue; involves D-Bus, cgroups.