Sudden Grub Error on update-grub

[tjh]

Hi,
I'm still runing Proxmox 7.4 - I haven't bitten the bullet for an 8 upgrade yet.

This morning I did an apt-get update / dist-upgrade (I am using the free repo) and I got a new kernel version:

Log started: 2023-07-31  10:45:04
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 52779 files and directories currently installed.)
Preparing to unpack .../linux-libc-dev_5.10.179-3_amd64.deb ...
Unpacking linux-libc-dev:amd64 (5.10.179-3) over (5.10.179-2) ...
Preparing to unpack .../pve-kernel-5.15.108-1-pve_5.15.108-2_amd64.deb ...
Unpacking pve-kernel-5.15.108-1-pve (5.15.108-2) over (5.15.108-1) ...
Setting up linux-libc-dev:amd64 (5.10.179-3) ...
Setting up pve-kernel-5.15.108-1-pve (5.15.108-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.108-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.15.108-1-pve /boot/vmlinuz-5.15.108-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done

Those grub errors - they don't look good.
Every time I re-run grub-update manually I get them:

root@max:/var/log/apt# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.108-1-pve
Found initrd image: /boot/initrd.img-5.15.108-1-pve
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
/usr/sbin/grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done

root@max:/var/log/apt# lvdisplay pve/root
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh
  LV Write Access        read/write
  LV Creation host, time proxmox, 2020-08-18 09:33:10 +1200
  LV Status              available
  # open                 1
  LV Size                59.25 GiB
  Current LE             15168
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

root@max:/var/log/apt# grub-probe --target=fs_uuid --device /dev/mapper/pve-root
grub-probe: error: disk `lvmid/3sZByl-ro2W-6hq3-2w3a-u37j-t03H-rD91fc/rBVg9c-QZjS-INpU-KsXn-TzTg-dyJm-jPATTh' not found.

I've done a bit of searching of the forums and the "fix" seem to be do to an lvm extend by 1g, then remove the 1g again.
But that seemed to apply to older versions of grub. I have version 2.06-3~deb11u5

Am I right to assume it's very unsafe to reboot until I resolve this issue?

Tim

 

[tjh]

hmm ok I seem to have fixed this by the "update something on the lvm" thing, so I just created a new VM with a 1G disk attached to that LVM and it seems to have fixed it.

 

[fweber]

Just for the sake of completeness: This issue is indeed caused by a grub bug [1], and package grub2 2.06-3 is still affected. Very likely, it would have grub to error out with "disk not found" on the next boot, as described here [2]. The workaround is indeed to change something in LVM, for example by creating a new VM with a LVM thin volume as you did.

The bug is fixed in package grub2 2.06-13, which is available in Debian Bookworm [3], and thus, PVE 8.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987008
[2] https://pve.proxmox.com/wiki/Recover_From_Grub_Failure#Recovering_from_grub_"disk_not_found"_error_when_booting_from_LVM
[3] https://tracker.debian.org/pkg/grub2

 

[tjh]

Just for the sake of completeness:

Thank you so much for your followup. It puts my mind at ease that I've done the right thing and, while I don't plan to update to 8.0 just yet, when I get around to it it's good to know the bug is fixed there.

Really appreciate your detailed answer with pointers - thank you.

Tim