[SOLVED] Proxmox hangs at machine-id check during upgrade from 6 to 7

[Lymond]

We are upgrading a single node of our cluster from 6.4 to 7.3. We run a Debian install with Proxmox on top. Our two test systems went fine, but this latest system, with a Supermicro motherboard X9DRW-iF, first hung at 99%. Upgrade ran for about 45 minutes then stopped at 99% for an hour before we killed the process. Running dpkg --configure -a gets to "machine-ID check OK" and nothing more.

Our current pveversion shows most modules updated but pve-manager and proxmox-ve are both "not correctly installed.

pveversion -v
proxmox-ve: not correctly installed (running kernel: 5.15.83-1-pve)
pve-manager: not correctly installed (running version: 7.3-4/d69b70d4)
pve-kernel-5.15: 7.3-1
pve-kernel-helper: 7.3-1
pve-kernel-5.4: 6.4-20
pve-kernel-5.3: 6.1-6
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.1-1
proxmox-backup-file-restore: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve1

These are the last few lines of the terminal before the hang at 99%.

tail term.log
 ==> Package distributor has shipped an updated version.
 ==> Keeping old config file as default.
Installing new version of config file /etc/vzdump.conf ...
Created symlink /etc/systemd/system/multi-user.target.wants/pvescheduler.service → /lib/systemd/system/pvescheduler.service.
machine-id check OK
Backing up lvm.conf before setting pve-manager specific settings..
'/etc/lvm/lvm.conf' -> '/etc/lvm/lvm.conf.bak'
Setting 'global_filter' in /etc/lvm/lvm.conf to prevent zvols from being scanned:
global_filter=["a|.*|"] => global_filter=["r|/dev/zd.*|"]

We do have an NFS backend server for image storage over infiniband as well as a samba connection for VM backups over ethernet. The fiber connection to the rest of the cluster and backend was not connected during the upgrade, nor was the SMB connection.

Looking for ideas on steps forward. Thank you.

 

[Moayad]

Hello,

May I ask how you did the upgrade? I mean, have you used apt update && apt upgrade or apt update && apt dist-upgrade?

I would check if the Server have enough space on the disk and the RAM usage. Can you also check the system logs syslog/journalctl to see if there are any errors or messages that could provide more information about the cause of the issue?

 

[Lymond]

Solution: once we booted all the cluster nodes on the new fiber switch and all were talking over inifiniband, we retried dpkg --configure -a and it completed in 2 seconds. After that, all the upgrades to the other VM nodes went perfectly. So I think if we'd not tested the new inifiniband card off-cluster, and not running the upgrade off-cluster, we would have been fine.

 

[Moayad]

Hi,

Glad that you solved the issue yourself!

I will go ahead and set your thread as [SOLVED] to help other people who have the same issue.

Have a good day