update-grub hanging

[Jeremy Andrews]

I'm unable to apply updates on my Proxmox server. Currently running:

Linux pve 5.0.21-1-pve #1 SMP PVE 5.0.21-2 (Wed, 28 Aug 2019 15:12:18 +0200) x86_64 GNU/Linux

Attempts to apply updates fail as follow:

root@pve:/boot/grub# apt update
Hit:1 http://download.proxmox.com/debian buster InRelease                     
Hit:2 http://security.debian.org buster/updates InRelease                     
Hit:3 http://download.proxmox.com/debian/corosync-3 stretch InRelease         
Hit:4 http://ftp.it.debian.org/debian buster InRelease                         
Hit:5 https://download.docker.com/linux/debian buster InRelease     
Hit:6 http://ftp.it.debian.org/debian buster-updates InRelease           
Hit:7 https://enterprise.proxmox.com/debian/pve buster InRelease         
Reading package lists... Done
Building dependency tree       
Reading state information... Done
20 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@pve:/boot/grub# apt upgrade
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

Following the instructions simply hangs (I've left it >12 hours, overnight until I finally cancel it):

root@pve:/boot/grub# dpkg --configure -a
Setting up pve-kernel-5.0.21-2-pve (5.0.21-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/dkms 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
dkms: WARNING: Linux headers are missing, which may explain the above failures.
      please install the linux-headers-5.0.21-2-pve package to fix this.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
update-initramfs: Generating /boot/initrd.img-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/pve-auto-removal 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-pve-efiboot 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve

Doing some web searches pointed to possibly an issue with `update-grub`, so I've also tried simply running that command itself, and it also hangs (I've left it running >12 hours overnight, nothing visible is happening):

root@pve:/# update-grub

 

[Jeremy Andrews]

Here's `strace` output when running `update-grub`

 

[fabian]

can you try running

grub-probe --target=device /
grub-probe --target=device /boot

and then

grub-probe --device OUTPUT_FROM_ABOVE --target=fs_uuid

for all of the devices output by the first two commands? please include all the output in your post!

 

[Jeremy Andrews]

Both of those commands seem to be hanging also.

Another command that seems to be hanging now is:

zpool status

 

[Jeremy Andrews]

I now suspect this was originally caused by an attempt to mount a USB drive on the Proxmox server some days ago. Attached is the dmesg output when that failed.

 

[fabian]

what kind of disks are you using for your zpools? any single-device zpools?

 

[Jeremy Andrews]

I have 3 pools currently mounted, none of them are single-device zpools.

The boot pool is built from 6 x 2.5" SSDs.

Another pool is made up of 3 x high-speed SSDs.

The largest pool is made up of 8 x 3.5" spinning drives.

The fourth pool which failed to mount was made of of 6 x 3.5" drives (and is in an external USB 3 device)

 

[fabian]

I suggest rebooting the host (given the current state, it is likely that you the shutdown will not finish cleanly and you need some sort of reset), and attempting to finish the upgrade afterwards.

 

[Jeremy Andrews]

As expected, I had to reset -- shutdown successfully unmounted all three mounted zfs pools, then it hung during the final shutdown process. I held down the power button to force the server to shut off, then rebooted.

Now the server will not boot, offering the following error:

Reboot and Select proper Boot device
or Insert Boot Media in seleted Boot device and press a key...

 

[fabian]

As expected, I had to reset -- shutdown successfully unmounted all three mounted zfs pools, then it hung during the final shutdown process. I held down the power button to force the server to shut off, then rebooted.

Now the server will not boot, offering the following error:

Reboot and Select proper Boot device
or Insert Boot Media in seleted Boot device and press a key...

that sounds like a hardware problem? maybe try a cold boot? otherwise check which disks/hbas/.. are visible to the bios for booting.

 

[Jeremy Andrews]

Yeah, sorry -- I had replaced a failed hard drive while the server was off, bad timing. I returned the failed drive to the pool and it booted fine. I'll track that down later.

The server booted well, and then I was able to apply updates without any problems. Sorry for the noise.