update-grub hanging

Oct 9, 2019
7
1
8
50
I'm unable to apply updates on my Proxmox server. Currently running:
Bash:
Linux pve 5.0.21-1-pve #1 SMP PVE 5.0.21-2 (Wed, 28 Aug 2019 15:12:18 +0200) x86_64 GNU/Linux

Attempts to apply updates fail as follow:
Bash:
root@pve:/boot/grub# apt update
Hit:1 http://download.proxmox.com/debian buster InRelease                     
Hit:2 http://security.debian.org buster/updates InRelease                     
Hit:3 http://download.proxmox.com/debian/corosync-3 stretch InRelease         
Hit:4 http://ftp.it.debian.org/debian buster InRelease                         
Hit:5 https://download.docker.com/linux/debian buster InRelease     
Hit:6 http://ftp.it.debian.org/debian buster-updates InRelease           
Hit:7 https://enterprise.proxmox.com/debian/pve buster InRelease         
Reading package lists... Done
Building dependency tree       
Reading state information... Done
20 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@pve:/boot/grub# apt upgrade
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

Following the instructions simply hangs (I've left it >12 hours, overnight until I finally cancel it):
Bash:
root@pve:/boot/grub# dpkg --configure -a
Setting up pve-kernel-5.0.21-2-pve (5.0.21-3) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/dkms 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
dkms: WARNING: Linux headers are missing, which may explain the above failures.
      please install the linux-headers-5.0.21-2-pve package to fix this.
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
update-initramfs: Generating /boot/initrd.img-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/pve-auto-removal 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
run-parts: executing /etc/kernel/postinst.d/zz-pve-efiboot 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.0.21-2-pve /boot/vmlinuz-5.0.21-2-pve

Doing some web searches pointed to possibly an issue with `update-grub`, so I've also tried simply running that command itself, and it also hangs (I've left it running >12 hours overnight, nothing visible is happening):
Bash:
root@pve:/# update-grub
 
can you try running
Code:
grub-probe --target=device /
grub-probe --target=device /boot

and then
Code:
grub-probe --device OUTPUT_FROM_ABOVE --target=fs_uuid

for all of the devices output by the first two commands? please include all the output in your post!
 
I now suspect this was originally caused by an attempt to mount a USB drive on the Proxmox server some days ago. Attached is the dmesg output when that failed.
 

Attachments

  • usb-error.txt
    10.9 KB · Views: 4
what kind of disks are you using for your zpools? any single-device zpools?
 
I have 3 pools currently mounted, none of them are single-device zpools.

The boot pool is built from 6 x 2.5" SSDs.

Another pool is made up of 3 x high-speed SSDs.

The largest pool is made up of 8 x 3.5" spinning drives.

The fourth pool which failed to mount was made of of 6 x 3.5" drives (and is in an external USB 3 device)
 
I suggest rebooting the host (given the current state, it is likely that you the shutdown will not finish cleanly and you need some sort of reset), and attempting to finish the upgrade afterwards.
 
As expected, I had to reset -- shutdown successfully unmounted all three mounted zfs pools, then it hung during the final shutdown process. I held down the power button to force the server to shut off, then rebooted.

Now the server will not boot, offering the following error:
Code:
Reboot and Select proper Boot device
or Insert Boot Media in seleted Boot device and press a key...
 
As expected, I had to reset -- shutdown successfully unmounted all three mounted zfs pools, then it hung during the final shutdown process. I held down the power button to force the server to shut off, then rebooted.

Now the server will not boot, offering the following error:
Code:
Reboot and Select proper Boot device
or Insert Boot Media in seleted Boot device and press a key...

that sounds like a hardware problem? maybe try a cold boot? otherwise check which disks/hbas/.. are visible to the bios for booting.
 
Yeah, sorry -- I had replaced a failed hard drive while the server was off, bad timing. I returned the failed drive to the pool and it booted fine. I'll track that down later.

The server booted well, and then I was able to apply updates without any problems. Sorry for the noise.
 
  • Like
Reactions: fabian

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!