🛟 Troubleshooting Linux Boot Failures: GRUB and initramfs

A server that hangs at boot is a different kind of stress than a service that’s down. You can’t SSH in, your usual tools are gone, and you’re staring at a half-rendered console or a grub> prompt with no idea where to start. I’ve sat in front of plenty of these, and the thing that turns panic into a 10-minute fix is having a map of the boot sequence and knowing which stage you’re stuck in. Let’s build that map.

The boot sequence, so you can locate the failure

You can only fix a boot failure if you know which stage failed:

1. Firmware (BIOS/UEFI) hands off to the bootloader.
2. GRUB loads its config, shows the menu, and loads the kernel + initramfs.
3. Kernel initializes and mounts the initramfs (a tiny in-memory root).
4. initramfs finds and mounts the real root filesystem, then pivots to it.
5. systemd (PID 1) takes over and brings up the rest of userspace.

Each stage fails differently, and the symptom tells you where you are. A grub> prompt is stage 2. “Kernel panic - not syncing: VFS: Unable to mount root fs” is stage 4. An (initramfs) prompt is stage 4. An emergency mode shell is stage 5. Identify the stage first; don’t start randomly editing things.

Stuck at the GRUB prompt (stage 2)

If you get a bare grub> or grub rescue>, GRUB loaded but couldn’t find or read its config. You can boot manually from the prompt:

grub> ls                      # list partitions, e.g. (hd0,gpt2)
grub> ls (hd0,gpt2)/          # find the one with /boot
grub> set root=(hd0,gpt2)
grub> linux /vmlinuz root=/dev/sda2
grub> initrd /initrd.img
grub> boot

Once you’re booted, fix it permanently so it doesn’t recur:

sudo grub-install /dev/sda          # reinstall to the disk's boot sector
sudo update-grub                    # Debian/Ubuntu
# or: sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL

Editing a single boot, non-destructively

You don’t have to commit a config change to test one. At the GRUB menu, press e to edit the selected entry in memory, change the kernel line, and Ctrl-X to boot it once. Nothing is saved — reboot and you’re back to the original. This is the safe way to test kernel parameters.

The two parameters worth knowing:

• Append systemd.unit=rescue.target to get a minimal single-user shell, or emergency.target for an even more minimal one — useful when a failing service blocks normal boot.
• Append init=/bin/bash to bypass systemd entirely and drop straight to a root shell (root mounted read-only). The nuclear option for “I locked myself out of everything.”

Dropped into the initramfs shell (stage 4)

An (initramfs) prompt means the kernel booted but couldn’t mount your real root filesystem. The usual culprits: a wrong root= UUID, a filesystem that needs checking, or a missing storage/RAID/LVM driver in the initramfs.

Find and check the root device from here:

(initramfs) cat /proc/cmdline        # what root= did the kernel get?
(initramfs) ls /dev/disk/by-uuid/    # do the UUIDs match fstab?
(initramfs) fsck -y /dev/sda2        # repair an unclean filesystem
(initramfs) exit                     # try to continue booting

A surprising number of initramfs hangs are just an unclean filesystem that needs fsck because the box lost power mid-write. Run it, exit, and you’re often back.

If a kernel update left you with a broken or incomplete initramfs, rebuild it from a working system or a chroot:

# Debian/Ubuntu
sudo update-initramfs -u -k all
# RHEL/Fedora
sudo dracut --force --regenerate-all

The rescue workflow: chroot from live media

When you can’t boot at all, boot a live USB or your cloud provider’s rescue mode, then chroot into the broken system to repair it as if it were running:

sudo mount /dev/sda2 /mnt              # your root partition
sudo mount /dev/sda1 /mnt/boot         # if /boot is separate
for d in dev proc sys run; do sudo mount --bind /$d /mnt/$d; done
sudo chroot /mnt
# Now you're "inside" the broken system:
update-grub && update-initramfs -u -k all
exit

This is the single most useful recovery technique on Linux, and it works for almost everything — fixing GRUB, rebuilding initramfs, resetting a forgotten root password, repairing fstab. Memorize the bind-mount-and-chroot dance.

Don’t forget fstab and emergency mode (stage 5)

If you reach an emergency mode prompt, you booted far enough that systemd is running — usually a bad /etc/fstab entry. A mistyped mount, a removed disk still listed, or a missing nofail option will hold the whole boot hostage. Read the journal for the failing unit:

journalctl -xb            # this boot's log, with the failure highlighted
systemctl --failed        # which units failed

Comment out the offending fstab line (add nofail for non-critical mounts so a missing disk never blocks boot again) and reboot.

Where AI helps under pressure

Boot failures are high-stress and you’re working from a console with no copy-paste and no search engine in front of you — exactly when a clear head is hardest. From my phone or a second machine, I describe the exact symptom (“GRUB drops to rescue, ls shows hd0,gpt1 and hd0,gpt2”) and ask for the specific recovery steps. The model is good at mapping a symptom to a stage and giving the precise commands, which is far better than half-remembering the chroot bind-mount order at midnight. Keep a saved Linux recovery prompt so you’re not composing it during the outage.

The non-negotiable: read every command before you run it, especially fsck, grub-install, and anything touching a partition. Recovery commands are exactly the ones that can make a recoverable situation unrecoverable.

The mindset that fixes boots fast

Identify the stage from the symptom. Use e at the GRUB menu to test changes without committing them. Know the chroot-from-live-media dance cold. Reach for fsck and journalctl -xb early. Do those four things and a non-booting server stops being a heart-attack and becomes a procedure. More server-recovery guides are in the AI for Linux Admins collection.

Boot recovery commands can be destructive. Verify device names and read every fsck or grub-install command before running it, especially when working from rescue media.