Linux Boot Failure & Rescue Prompt

You are a senior Linux sysadmin who has rescued unbootable servers — sometimes over IPMI, sometimes from a live USB, sometimes via cloud serial console. You move carefully because the next mistake could mean reinstalling.

I will provide:
- What changed before the failure (kernel update, fstab edit, GRUB reinstall, package upgrade, partition resize, encryption changes)
- The visible failure mode:
  - GRUB prompt (`grub>` rescue) — bootloader broken
  - `Cannot mount /` / `dropping to initramfs shell` — root not findable
  - Kernel panic `not syncing: VFS: Unable to mount root fs` — initramfs missing drivers
  - `Emergency mode` / `Failed to mount /var` — fstab issue at boot
  - Black screen / `No bootable device` — bootloader gone
- Screenshot or text of the actual console output (paste it)
- Boot mode (BIOS or UEFI)
- Disk layout (`/`, `/boot`, `/boot/efi` — separate partitions or not? LVM? LUKS-encrypted? mdraid?)
- Distro version and kernel (if known)
- Access available: BIOS console / IPMI / cloud serial / live USB / chroot from another OS

Your job:

1. **Identify the failure stage** in the boot chain:
   - **Firmware → bootloader missing** (no GRUB) — UEFI boot entry gone, or `/boot/efi` not mounted
   - **Bootloader → kernel missing** (GRUB present, can't find kernel) — `/boot` corruption, kernel files removed
   - **Kernel → initramfs missing/wrong** — driver missing for root device (common after kernel update without `dracut`/`update-initramfs`)
   - **Initramfs → root mount failure** — wrong root UUID in cmdline, LVM not assembling, LUKS not unlocking
   - **Root mounted → systemd init failure** — `fstab` line failing, network mount required for boot, dependency cycle to `local-fs.target`
2. **For each stage** give the minimal command sequence to confirm or recover, with the order that minimizes risk of making it worse:
   - Read-only inspect first (mount, check, identify) before write actions (reinstall, regenerate, edit).
   - Always `chroot` properly with `/dev`, `/proc`, `/sys` bind-mounts before running distro-aware commands like `update-grub` / `grub2-mkconfig`.
   - Reinstall GRUB only after confirming the target partition. On UEFI, this is the EFI system partition.
3. **If the user is at the `(initramfs)` shell**, walk them through:
   - `cat /proc/cmdline` to see what root device the kernel was told to use
   - `ls /dev/disk/by-uuid/` to find actual UUIDs
   - `vgchange -ay` / `cryptsetup luksOpen` if LVM/LUKS
   - `mount -o ro` first, then `exit` to continue boot or remount RW for fixes
4. **For fstab failures**: temporarily add `nofail,x-systemd.device-timeout=10` to non-critical entries, comment out the problem line, regenerate initramfs if needed.
5. **For kernel update gone wrong**: boot the previous kernel from GRUB menu, then investigate why the new initramfs is missing drivers (`dracut --regenerate-all -f` / `update-initramfs -u -k all`).
6. **Mark every DESTRUCTIVE step**: `grub-install` to wrong device, `mkfs` to wrong partition, `dd` to `/dev/sda` (vs `/dev/sda1`), removing the only working kernel.

---

Failure mode (paste exact screen text): [DESCRIBE / PASTE]
What changed immediately before: [DESCRIBE]
Boot mode: [BIOS / UEFI]
Disk layout: [/ on /dev/...; /boot on ...; LVM yes/no; LUKS yes/no; mdraid yes/no]
Distro: [e.g., Ubuntu 22.04 / RHEL 9 / Debian 12]
Access available: [IPMI / cloud serial / live USB / hypervisor console]
Output from where you are now (initramfs shell, GRUB prompt, live USB chroot):
```
[PASTE]
```

Why this prompt works

Boot failures are high-stakes and irreversible if you guess. Half the rescues fail because someone reinstalled GRUB to the wrong device, or edited /etc/fstab from a broken root mount and made it worse. This prompt enforces a stage-aware walk: identify which stage failed, then act — read-only first, write-actions last.

How to use it

1. Don’t panic-reboot. Each reboot loses the on-screen evidence and can take you further from a fixable state.
2. Capture the exact screen text. Phone photos work; transcribe key lines.
3. Know your access path: IPMI, cloud serial console, hypervisor console, live USB. Each enables different recovery steps.
4. Always disclose what you changed. “It just stopped working” is almost never true — a kernel update, fstab edit, or package upgrade is upstream.

Recovery cheatsheet by stage

Stage 1: No GRUB (“No bootable device” / black screen)

# Boot a live USB of the same distro family
# Identify partitions
lsblk -f
# UEFI: identify EFI partition (usually FAT, ~512MB)
# BIOS: identify boot device (the one in BIOS boot order)

# Mount root and chroot
sudo mount /dev/mapper/vg-root /mnt           # adjust to your /
sudo mount /dev/sda2 /mnt/boot                # if separate /boot
sudo mount /dev/sda1 /mnt/boot/efi            # if UEFI
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run
sudo mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars  # UEFI only
sudo chroot /mnt

# Inside the chroot, reinstall bootloader
# UEFI (Ubuntu/Debian):
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu
update-grub
# UEFI (RHEL/Fedora):
grub2-install --target=x86_64-efi --efi-directory=/boot/efi
grub2-mkconfig -o /boot/efi/EFI/<distro>/grub.cfg

# BIOS (Ubuntu/Debian):
grub-install /dev/sda          # the DISK, not a partition
update-grub
# BIOS (RHEL/Fedora):
grub2-install /dev/sda
grub2-mkconfig -o /boot/grub2/grub.cfg

exit                           # leave chroot
# Unmount in reverse order, then reboot

Stage 2: GRUB present but kernel won’t boot

• At the GRUB menu, press e on the entry. Try booting an older kernel from “Advanced options.”
• If the only kernel is the broken one, boot from live USB and reinstall the previous kernel.

Stage 3: (initramfs) shell

# Inside the initramfs busybox shell:
cat /proc/cmdline               # see what root= the kernel expected
ls /dev/disk/by-uuid/           # see what's actually present
blkid                           # filesystem types and UUIDs

# LVM not assembled?
vgchange -ay
lvs

# LUKS-encrypted root?
cryptsetup luksOpen /dev/sda3 cryptroot
vgchange -ay

# Try a manual mount
mkdir -p /sysroot
mount -o ro /dev/mapper/vg-root /sysroot

# If success: exit to continue boot, or remount rw for fixes
exit

Stage 4: Emergency mode (root mounted, systemd init failed)

# At login prompt, enter root password
systemctl --failed                       # which unit failed
journalctl -xb -p err                    # boot-time errors
journalctl -u <failed-unit>              # specific unit

# Common fix: a non-essential /etc/fstab line failing
mount -o remount,rw /
nano /etc/fstab                          # add `nofail,x-systemd.device-timeout=10` to problem entry
systemctl daemon-reload
systemctl reboot

Stage 5: Kernel panic — VFS Unable to mount root fs

Usually means the initramfs is missing the driver for the root device (e.g., after a kernel update on a VM with virtio or LVM, where update-initramfs failed or wasn’t run).

# From a live USB, chroot as in Stage 1, then:
# Debian/Ubuntu:
update-initramfs -u -k all
# Fedora/RHEL:
dracut --regenerate-all --force

# Verify the initramfs has the right modules
lsinitramfs /boot/initrd.img-<ver> | grep -E "virtio|nvme|lvm"

Common findings this catches

• GRUB rescue prompt after apt full-upgrade — /boot filled up; new initramfs couldn’t write. Free space, regenerate initramfs.
• Cannot mount /home blocks boot — /etc/fstab entry without nofail for a non-critical mount. Add nofail, regenerate.
• device not found UUID=... in initramfs — UUID changed after partition resize; update /etc/fstab and /etc/default/grub GRUB_CMDLINE_LINUX.
• UEFI boots straight to firmware (no GRUB menu) — efibootmgr entries gone; recreate from a chroot.
• Kernel panic after kernel update on virtio VM — initramfs missing virtio_blk; regenerate with update-initramfs -u.
• LUKS prompt never appears, system hangs at GRUB — cryptdevice= or rd.luks.uuid= parameter wrong in cmdline.

Boot debugging tricks

Add to GRUB cmdline temporarily (press e at GRUB menu, edit the linux line):

nomodeset             # if framebuffer crashes
single                # boot to single-user shell
init=/bin/bash rw     # bypass systemd ENTIRELY (last resort; no network, no logging)
systemd.unit=rescue.target          # systemd rescue mode (root password required)
systemd.unit=emergency.target       # even more minimal
systemd.debug-shell=1               # enable debug shell on Alt+F9
rd.break                            # break in initramfs BEFORE root mount

When to escalate

• Hardware failure indicators (dmesg from live USB shows disk I/O errors) — replace the disk; rescue may be futile.
• LUKS recovery without the passphrase or header backup — call security team; data recovery is now a project.
• Production server with no IPMI/serial console — coordinate with data-center for physical access before improvising.
• Cloud VM unbootable without a serial console option — most providers offer “rescue mode” / “recovery boot”; use that rather than DIY chroot.