So I'm not sure exactly what sequence of events or circumstances caused an issue, but a recent update to PVE sent both my kernel and me into a panic.
TLDR, a few days ago I did a minor upgraded from 8.1.? (kernel 6.5.13) to 8.1.? (kernel 6.5.15). Apparently, in this upgrade, my grub configuration changed such that my
I THINK I rebooted without issue (not 100% sure), but today I saw the new upgrade to 8.2.2 and just pressed go. Then on reboot I was unable to boot into either the 6.5.15 or the new 6.8.4(?) kernel from the grub menu. I was fortunately able to boot back into the 6.5.13 version (still had root=GUID=... in the grub config).
What happened?!?
A bit more detail.
(...and yes now I know that the i915 GUC stuff wouldn't wouldn't have worked on 6.8 so I'd have needed roll back kernels anyway...)
I already lost the old one since I hacked my way back to a grub.cfg that didn't have them. But basically I think it had `root=/dev/sdi1` followed by a newline, a tab and then `/dev/sdj1 ro` and then the rest. I'm not 100% sure but I'm pretty sure both members of that mirror were referenced.
The message on boot mentioned something like "unable to execute command /dev/sdj1", implying that it the \n\t in there was interpreted as a separate command. So maybe it's not the GUID change that's at issue, but the addition of the newline?
Anyway, I'm not down and out, but I'm quite curious as to what could have happened and if there's anything I can do to prevent this in the future.
TLDR, a few days ago I did a minor upgraded from 8.1.? (kernel 6.5.13) to 8.1.? (kernel 6.5.15). Apparently, in this upgrade, my grub configuration changed such that my
linux command went from referencing the root partition by GUID to using its /dev/sd? mapping. And this led to being unable to find the boot partition.I THINK I rebooted without issue (not 100% sure), but today I saw the new upgrade to 8.2.2 and just pressed go. Then on reboot I was unable to boot into either the 6.5.15 or the new 6.8.4(?) kernel from the grub menu. I was fortunately able to boot back into the 6.5.13 version (still had root=GUID=... in the grub config).
What happened?!?
A bit more detail.
- CPU: 24 x 13th Gen Intel(R) Core(TM) i7-13700K (1 Socket)
- RAM: 128 GB
- Boot Mode: efi
- "Root" Disk: BTRFS mirror (2x1TB SATA SSD)
linux /boot/vmlinuz-6.5.13-3-pve root=UUID=ed4c1232-afb0-45a4-8b9e-4de2505d606b ro intel_iommu=on i915.enable_guc=3 i915.max_vfs=7 iommu=pt quiet(...and yes now I know that the i915 GUC stuff wouldn't wouldn't have worked on 6.8 so I'd have needed roll back kernels anyway...)
I already lost the old one since I hacked my way back to a grub.cfg that didn't have them. But basically I think it had `root=/dev/sdi1` followed by a newline, a tab and then `/dev/sdj1 ro` and then the rest. I'm not 100% sure but I'm pretty sure both members of that mirror were referenced.
The message on boot mentioned something like "unable to execute command /dev/sdj1", implying that it the \n\t in there was interpreted as a separate command. So maybe it's not the GUID change that's at issue, but the addition of the newline?
Anyway, I'm not down and out, but I'm quite curious as to what could have happened and if there's anything I can do to prevent this in the future.