PVE 6.4-15 to 7 upgrade hanging

[Blazemonkey]

Hey folks, I'm having some issues upgrading PVE to version 7 on this system. It seems to be hanging at 58% during the upgrade.

Here's where it's getting stuck:

Found Debian GNU/Linux 10 (buster) on /dev/mapper/pve-vm--115--disk--0
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
File descriptor 3 (pipe:[188629]) leaked on vgs invocation. Parent PID 30125: /usr/sbin/grub-probe
/usr/sbin/grub-probe: error: disk `lvmid/VvrFeF-HkQ9-3Zp8-Dy7e-18Kc-D88V-ZLcB46/G7GZdl-BZq2-dw9A-3K8W-kPmZ-OGtF-SRVTcr'                                                                not found.
Found Debian GNU/Linux 10 (buster) on /dev/mapper/pve-vm--130--disk--0
Adding boot menu entry for UEFI Firmware Settings ...
done

Progress: [ 58%] [#############################################################################################....................................................................]

In my limited ability to troubleshoot kernel/upgrade issues, I opened another terminal to check, and it looks like it's getting hung up on a memtest86 package...

root@Proxmox-Server:~# ps af
  PID TTY      STAT   TIME COMMAND
30509 pts/2    Ss     0:00 -bash
  945 pts/2    R+     0:00  \_ ps af
 8419 pts/0    Ss     0:00 -bash
 6954 pts/0    S+     0:01  \_ apt dist-upgrade
27585 pts/1    Ss+    0:00      \_ /usr/bin/dpkg --status-fd 23 --configure --pending
27586 pts/1    S+     0:00          \_ /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/memtest86+.postin
27595 pts/1    Z+     0:00              \_ [memtest86+.post] <defunct>
 2308 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root@Proxmox-Server:~#

I have plenty of space to work with as well.

root@Proxmox-Server:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   16G     0   16G   0% /dev
tmpfs                 3.2G  1.1M  3.2G   1% /run
/dev/mapper/pve-root   59G  4.2G   51G   8% /
tmpfs                  16G   31M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  16G     0   16G   0% /sys/fs/cgroup
/dev/sdi2             511M  312K  511M   1% /boot/efi
zpool-content         899G  1.7G  898G   1% /zpool-content
zpool-backup          429G  151G  279G  35% /zpool-backup
/dev/fuse              30M   16K   30M   1% /etc/pve
tmpfs                 3.2G     0  3.2G   0% /run/user/0

Rebooting the box at this point will kill it.. Any suggestions on how to move forward?

 

[Blazemonkey]

Anyways, I killed PID 27586 and the upgrade appeared to complete

I saw the errors:

dpkg: error processing package memtest86+ (--configure):

 installed memtest86+ package post-installation script subprocess was killed by signal (Terminated)

and

Errors were encountered while processing:

 memtest86+

E: Sub-process /usr/bin/dpkg returned an error code (1)

And then ran apt-upgrade again, with no errors. I rebooted and the system seems fine. Just thought I'd thow this out there in case someone else has the same problem.

 

[Neobin]

Can you please provide the full output in code-tags of: cat /etc/default/grub
Just curious.

 

[Blazemonkey]

I did actually run an 'update-grub' after the upgrade completed, But here it is:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

 

[Neobin]

Thanks. Can you please also provide the output of: cat /etc/default/grub.d/proxmox-ve.cfg

I am no expert at all and maybe I am on the completely wrong track, but your very first output made me think, that the grub os-prober goes through all your VM-block devices and this is something you do not want: [1].

[1] https://forum.proxmox.com/threads/warning-os-prober.115087/#post-498072

 

[Blazemonkey]

Hmm, that did come up in my queries..

root@Proxmox-Server:~# cat /etc/default/grub.d/proxmox-ve.cfg
GRUB_DISTRIBUTOR="Proxmox VE"
GRUB_DISABLE_OS_PROBER=true

 

[Blazemonkey]

It's a very basic system tho, just a single node, with PVE installed directly to an SSD and a handful of containers. Nearly stock config from the default proxmox config, the only thing modified on the host was installing NFS-Kernel.

 

[Neobin]

GRUB_DISABLE_OS_PROBER=true

At least it seems to be disabled now. Maybe it was not before the upgrade or, as I said, I am completely on the wrong track.

Hopefully someone with more knowledge than me can shed some light in here, sorry.

 

[Blazemonkey]

It's hard to say, I wouldn't have thought to check that anyway. It's all good, I did manage to complete the update successfully and reboot, everything seems good, so I'm happy

 

[Blazemonkey]

For some reason the update got stuck on that memtest process. I'm not sure why, but maybe it was waiting on a user input that wasn't visible in the terminal.

 

[Neobin]

Fingers crossed that it will not happen again; but to be sure, it would still be good if someone (e.g. from the Staff) could have a look at it.
Might have to wait/see.

 

[Blazemonkey]

For sure! It would be nice to know. I don't see myself having to do any more version 6 to 7 upgrades again, thankfully!


I've been using Proxmox since version 1.9 and this was the first time I ever ran into trouble updating. This upgrade was actually coming from a brand new 6.4 install from the day prior though, not an ancient system

 

[johnhegs]

Anyways, I killed PID 27586 and the upgrade appeared to complete

I saw the errors:

dpkg: error processing package memtest86+ (--configure):

 installed memtest86+ package post-installation script subprocess was killed by signal (Terminated)

and

Errors were encountered while processing:

 memtest86+

E: Sub-process /usr/bin/dpkg returned an error code (1)

And then ran apt-upgrade again, with no errors. I rebooted and the system seems fine. Just thought I'd thow this out there in case someone else has the same problem.

Thanks for posting this with the "kill the memtest" work around. Saved me a heap of work upgrading my cluster from 6 to 7. Cheers.

 

[droid105]

Anyways, I killed PID 27586 and the upgrade appeared to complete

I saw the errors:

dpkg: error processing package memtest86+ (--configure):

 installed memtest86+ package post-installation script subprocess was killed by signal (Terminated)

and

Errors were encountered while processing:

 memtest86+

E: Sub-process /usr/bin/dpkg returned an error code (1)

And then ran apt-upgrade again, with no errors. I rebooted and the system seems fine. Just thought I'd thow this out there in case someone else has the same problem.

Thank you for this solution!

My upgrade froze at the same place and I was able to finish following your steps.

 

[joge]

Thanks for the hint. Was the same for me, I had to kill the pid.

NOTE: You have to check the PID upfront! Check with ps af and use the PID from the process above [memtest86+.post].

 

[EmilioMoreno]

Same happened here. I am doing some initial test to migrate our environments, and i had to do exactly the same.
Regarding this

cat /etc/default/grub.d/proxmox-ve.cfg

This file does not exists on any on my servers; 4 clusters updated to pve6.4-15, but the file is not there.

Should i create such file before migrate another one?

 

[cray-]

Thanks very much for this. Had the same issue going 6.4-15 to 7.3-6 and upgrade appeared to stall at 57%. Killing memtest fixed it.

 

[Vince-0]

ran into this today, grub-update was finding possible bootable files in lxc containers. I killed the running grub process from another terminal and the upgrade continued.

 

[rolambert]

I ran into this today, killed grub using another ssh terminal worked like a charm.