32-bit Debian VM fails to boot after upgrading to PVE 8.2

Z

zorun

New Member
Jul 1, 2024
5
6
3
I run a VM with Debian 11 in 32-bit mode (i686 image): Linux guest 6.1.0-22-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) i686 GNU/Linux

It was working perfectly fine when the hypervisor was running PVE 8.1.0.

However, after upgrading the hypervisor to PVE 8.2.0, the VM fails to boot. It cannot find its root disk, and the kernel prints the following error on the console:

Code: 
virtio: device uses modern interface but does not have VIRTIO_F_VERSION_1

This seems to be a regression in Qemu: https://gitlab.com/qemu-project/qemu/-/issues/1983

Here is the apt upgrade log with package versions before/after the upgrade:

Code: 
Start-Date: 2024-07-01  17:08:39
Commandline: apt upgrade
Install: python3-pyvmomi:amd64 (6.7.1-4.1, automatic), proxmox-kernel-6.5.13-5-pve-signed:amd64 (6.5.13-5, automatic), proxmox-kernel-6.8:amd64 (6.8.8-2, automatic), linux-headers-6.1.0-22-common:amd64 (6.1.94-1, automatic), pve-esxi-import-tools:amd64 (0.7.1, automatic), proxmox-kernel-6.8.8-2-pve-signed:amd64 (6.8.8-2, automatic), proxmox-firewall:amd64 (0.4.2, automatic), linux-headers-6.1.0-22-amd64:amd64 (6.1.94-1, automatic)
Upgrade: pve-docs:amd64 (8.1.3, 8.2.2), libcurl4:amd64 (7.88.1-10+deb12u5, 7.88.1-10+deb12u6), pve-edk2-firmware-ovmf:amd64 (4.2023.08-2, 4.2023.08-4), libsmartcols1:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), udev:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libvpx7:amd64 (1.12.0-1+deb12u2, 1.12.0-1+deb12u3), proxmox-backup-restore-image:amd64 (0.6.0, 0.6.1), python3.11:amd64 (3.11.2-6, 3.11.2-6+deb12u2), libltdl7:amd64 (2.4.7-5, 2.4.7-7~deb12u1), proxmox-default-kernel:amd64 (1.0.1, 1.1.0), libcurl3-gnutls:amd64 (7.88.1-10+deb12u5, 7.88.1-10+deb12u6), open-iscsi:amd64 (2.1.8-1, 2.1.8-1.pve1), openssh-client:amd64 (1:9.2p1-2+deb12u2, 1:9.2p1-2+deb12u3), libgdk-pixbuf2.0-bin:amd64 (2.42.10+dfsg-1+b1, 2.42.10+dfsg-1+deb12u1), proxmox-widget-toolkit:amd64 (4.1.3, 4.2.3), libpve-rs-perl:amd64 (0.8.7, 0.8.9), util-linux-locales:amd64 (2.38.1-5, 2.38.1-5+deb12u1), libnss-myhostname:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), pve-firmware:amd64 (3.9-1, 3.12-1), tzdata:amd64 (2023c-5+deb12u1, 2024a-0+deb12u1), zfs-zed:amd64 (2.2.2-pve1, 2.2.4-pve1), libpam-systemd:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libarchive13:amd64 (3.6.2-1, 3.6.2-1+deb12u1), ssh:amd64 (1:9.2p1-2+deb12u2, 1:9.2p1-2+deb12u3), pve-qemu-kvm:amd64 (8.1.2-6, 9.0.0-3), libnvpair3linux:amd64 (2.2.2-pve1, 2.2.4-pve1), tar:amd64 (1.34+dfsg-1.2, 1.34+dfsg-1.2+deb12u1), proxmox-mail-forward:amd64 (0.2.2, 0.2.3), libproxmox-acme-perl:amd64 (1.5.0, 1.5.1), libpve-cluster-api-perl:amd64 (8.0.5, 8.0.7), pve-edk2-firmware-legacy:amd64 (4.2023.08-2, 4.2023.08-4), pve-ha-manager:amd64 (4.0.3, 4.0.5), libpve-apiclient-perl:amd64 (3.3.1, 3.3.2), lxcfs:amd64 (5.0.3-pve4, 6.0.0-pve2), libuutil3linux:amd64 (2.2.2-pve1, 2.2.4-pve1), libdav1d6:amd64 (1.0.0-2, 1.0.0-2+deb12u1), libpve-storage-perl:amd64 (8.0.5, 8.2.3), linux-compiler-gcc-12-x86:amd64 (6.1.67-1, 6.1.94-1), libsystemd0:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libfreetype6:amd64 (2.12.1+dfsg-5, 2.12.1+dfsg-5+deb12u3), libmount1:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), libcjson1:amd64 (1.7.15-1, 1.7.15-1+deb12u1), libzpool5linux:amd64 (2.2.2-pve1, 2.2.4-pve1), libpve-guest-common-perl:amd64 (5.0.6, 5.1.3), linux-headers-amd64:amd64 (6.1.67-1, 6.1.94-1), openssh-server:amd64 (1:9.2p1-2+deb12u2, 1:9.2p1-2+deb12u3), proxmox-kernel-6.5:amd64 (6.5.11-7, 6.5.13-5), libpython3.11-minimal:amd64 (3.11.2-6, 3.11.2-6+deb12u2), libgdk-pixbuf-2.0-0:amd64 (2.42.10+dfsg-1+b1, 2.42.10+dfsg-1+deb12u1), pve-cluster:amd64 (8.0.5, 8.0.7), libglib2.0-data:amd64 (2.74.6-2, 2.74.6-2+deb12u3), util-linux:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), less:amd64 (590-2, 590-2.1~deb12u2), util-linux-extra:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), systemd:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libudev1:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libpostproc56:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), linux-kbuild-6.1:amd64 (6.1.67-1, 6.1.94-1), fdisk:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), proxmox-ve:amd64 (8.1.0, 8.2.0), lxc-pve:amd64 (5.0.2-4, 6.0.0-1), grub-efi-amd64:amd64 (2.06-13+pmx1, 2.06-13+pmx2), libfdisk1:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), usbutils:amd64 (1:014-1, 1:014-1+deb12u1), proxmox-backup-file-restore:amd64 (3.1.2-1, 3.2.4-1), libc6:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), locales:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), libssl3:amd64 (3.0.11-1~deb12u2, 3.0.13-1~deb12u1), gstreamer1.0-x:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), usrmerge:amd64 (35, 37~deb12u1), ifupdown2:amd64 (3.2.0-1+pmx7, 3.2.0-1+pmx8), libpython3.11:amd64 (3.11.2-6, 3.11.2-6+deb12u2), libavcodec59:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libuuid1:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), qemu-server:amd64 (8.0.10, 8.2.1), libpve-access-control:amd64 (8.0.7, 8.1.4), bash:amd64 (5.2.15-2+b2, 5.2.15-2+b7), pve-container:amd64 (5.0.8, 5.1.12), libproxmox-acme-plugins:amd64 (1.5.0, 1.5.1), uuid-runtime:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), shim-helpers-amd64-signed:amd64 (1+15.7+1+pmx1, 1+15.8+1+pmx1), gstreamer1.0-plugins-base:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), pve-i18n:amd64 (3.1.5, 3.2.2), proxmox-offline-mirror-helper:amd64 (0.6.3, 0.6.6), base-files:amd64 (12.4+deb12u4, 12.4+deb12u6), grub-efi-amd64-signed:amd64 (1+2.06+13+pmx1, 1+2.06+13+pmx2), libpython3.11-stdlib:amd64 (3.11.2-6, 3.11.2-6+deb12u2), gnutls-bin:amd64 (3.7.9-2+deb12u1, 3.7.9-2+deb12u3), libavutil57:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libopeniscsiusr:amd64 (2.1.8-1, 2.1.8-1.pve1), libunbound8:amd64 (1.17.1-2+deb12u1, 1.17.1-2+deb12u2), libswscale6:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), proxmox-backup-client:amd64 (3.1.2-1, 3.2.4-1), libcryptsetup12:amd64 (2:2.6.1-4~deb12u1, 2:2.6.1-4~deb12u2), distro-info-data:amd64 (0.58+deb12u1, 0.58+deb12u2), mariadb-common:amd64 (1:10.11.4-1~deb12u1, 1:10.11.6-0+deb12u1), grub-efi-amd64-bin:amd64 (2.06-13+pmx1, 2.06-13+pmx2), grub2-common:amd64 (2.06-13+pmx1, 2.06-13+pmx2), libseccomp2:amd64 (2.5.4-1+b3, 2.5.4-1+deb12u1), libpve-http-server-perl:amd64 (5.0.5, 5.1.0), libc-dev-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), mount:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), libglib2.0-0:amd64 (2.74.6-2, 2.74.6-2+deb12u3), pve-manager:amd64 (8.1.3, 8.2.4), libpve-common-perl:amd64 (8.1.0, 8.2.1), openssh-sftp-server:amd64 (1:9.2p1-2+deb12u2, 1:9.2p1-2+deb12u3), libpve-network-perl:amd64 (0.9.5, 0.9.8), nano:amd64 (7.2-1, 7.2-1+deb12u1), libc-l10n:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), grub-common:amd64 (2.06-13+pmx1, 2.06-13+pmx2), python3.11-minimal:amd64 (3.11.2-6, 3.11.2-6+deb12u2), libmariadb3:amd64 (1:10.11.4-1~deb12u1, 1:10.11.6-0+deb12u1), libc-bin:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), libsystemd-shared:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), libswresample4:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libpve-notify-perl:amd64 (8.0.5, 8.0.7), libisl23:amd64 (0.25-1, 0.25-1.1), libc6-dev:amd64 (2.36-9+deb12u3, 2.36-9+deb12u7), libzfs4linux:amd64 (2.2.2-pve1, 2.2.4-pve1), systemd-sysv:amd64 (252.19-1~deb12u1, 252.26-1~deb12u2), python3-idna:amd64 (3.3-1, 3.3-1+deb12u1), libblkid1:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), libgnutls30:amd64 (3.7.9-2+deb12u1, 3.7.9-2+deb12u3), libavformat59:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), curl:amd64 (7.88.1-10+deb12u5, 7.88.1-10+deb12u6), pve-firewall:amd64 (5.0.3, 5.0.7), libgstreamer-plugins-base1.0-0:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), shim-signed:amd64 (1.39+pmx1+15.7-1+pmx1, 1.40+pmx1+15.8-1+pmx1), libgnutlsxx30:amd64 (3.7.9-2+deb12u1, 3.7.9-2+deb12u3), libpq5:amd64 (15.5-0+deb12u1, 15.7-0+deb12u1), bsdutils:amd64 (1:2.38.1-5+b1, 1:2.38.1-5+deb12u1), libgnutls-dane0:amd64 (3.7.9-2+deb12u1, 3.7.9-2+deb12u3), shim-signed-common:amd64 (1.39+pmx1+15.7-1+pmx1, 1.40+pmx1+15.8-1+pmx1), dns-root-data:amd64 (2023010101, 2024041801~deb12u1), bsdextrautils:amd64 (2.38.1-5+b1, 2.38.1-5+deb12u1), zfsutils-linux:amd64 (2.2.2-pve1, 2.2.4-pve1), shim-unsigned:amd64 (15.7-1+pmx1, 15.8-1+pmx1), postfix:amd64 (3.7.6-0+deb12u2, 3.7.11-0+deb12u1), openssl:amd64 (3.0.11-1~deb12u2, 3.0.13-1~deb12u1), proxmox-offline-mirror-docs:amd64 (0.6.3, 0.6.6), libpve-cluster-perl:amd64 (8.0.5, 8.0.7), linux-libc-dev:amd64 (6.1.67-1, 6.1.94-1), libavfilter8:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libgdk-pixbuf2.0-common:amd64 (2.42.10+dfsg-1, 2.42.10+dfsg-1+deb12u1)
End-Date: 2024-07-01  17:11:53
 
Hi,
seems like the upstream issue got closed, because a fix included in QEMU 8.2 fixed it for the maintainer. But maybe that fix is not working in all cases. Please share the VM configuration qm config 100 and output of lscpu on the host.
 
Code: 
Commandline: apt upgrade
P.S. Please always use full-upgrade or dist-upgrade. There are situations where just upgrade can lead to issues in Proxmox VE.
 
  • Like
Reactions: Kingneutron
Here is the VM config (without the workaround):

YAML: 
agent: 1
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: local:iso/debian-12.4.0-i386-netinst.iso,media=cdrom,size=641M
memory: 4096
meta: creation-qemu=8.1.2,ctime=1703869711
name: cfarm26
net0: virtio=BC:24:11:EE:4A:03,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,discard=on,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=b30a5e61-1725-40a4-8c68-fbb347c45a63
sockets: 1
vmgenid: e44af2c8-cb3c-4607-92dd-552a8d447691

And lscpu on the host:

Code: 
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   4
  On-line CPU(s) list:    0-3
Vendor ID:                GenuineIntel
  BIOS Vendor ID:         Intel
  Model name:             Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz
    BIOS Model name:      Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz Fill By OEM CPU @ 3.2GHz
    BIOS CPU family:      205
    CPU family:           6
    Model:                60
    Thread(s) per core:   1
    Core(s) per socket:   4
    Socket(s):            1
    Stepping:             3
    CPU(s) scaling MHz:   98%
    CPU max MHz:          3600.0000
    CPU min MHz:          800.0000
    BogoMIPS:             6385.67
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nons
                          top_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault pti tpr_shadow f
                          lexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt dtherm ida arat pln pts vnmi
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    128 KiB (4 instances)
  L1i:                    128 KiB (4 instances)
  L2:                     1 MiB (4 instances)
  L3:                     6 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-3
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          KVM: Mitigation: Split huge pages
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
  Mds:                    Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Unknown: No mitigations
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Vulnerable: No microcode
  Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
 
It seems like this won't be possible to fix in SeaBIOS in a way that makes everybody happy and might even be a bug in 32-bit Linux kernels with PAE enabled:
https://mail.coreboot.org/hyperkitt...org/message/6W6FNVJTE27QASARUKDSEWBDKB5ONKCY/
https://mail.coreboot.org/hyperkitt...org/message/PA5LH4PC53JCOAHQ3WYSNHOHFDDOHYLM/

Workarounds are using the lm=off CPU option or not using VirtIO devices or limiting the RAM for such guests (from my testing, you might need to go down to about 3 GiB with machine type i440fx, even lower with q35).
 
  • Like
Reactions: zorun and fireon
updated from proxmox 8.1 to 8.3 and hit this issue with one 32bit VM.
so the bug is still there.

disabled longmode and the VM now boots again.
thank you!


comparing lscpu, the difference is:

before
Code: 
Architecture:             i686
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual

after
Code: 
Architecture:             i686
  CPU op-mode(s):         32-bit
  Address sizes:          36 bits physical, 0 bits virtual


I wonder if there is any performance constraint with this.
probably not since 32bit is not using longmode at all?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom