Proxmox Virtual Environment 9.0 released!

[Bugsa]

Hello, I don't know if this is right place to report but I have issues with running OPNSense in KVM with 6.14 kernel. I had this problem even with PVE 8 with 6.14 kernel. When I switch back to kernel 6.8 everything works. I don't know how to debug it because there is not error in journalctl in host. OPNSense inside KVM starting to boot but then it says "bus error" and just cannot mount storage and continue to boot. I tried switch to UEFI, to Q35, tried all possible controllers and IDE/SATA/VirtIO/SCSI options to connect hard disk but neither is working with 6.14. Anyone having same issue?

PVE host CPU is: 4 x Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (1 Socket)

Edit: Even OPNSense install ISO cannot boot.

 

[alexantonov]

Congrats to the team on another huge milestone achieved!
I've seen there were plans to incorporate the ProxLB module into Proxmox VE v9.0. Has that been revised?

 

[fiona]

Hi,

Is anyone experiencing this issue or bug where, when you try to migrate a VM that is using LVM-thin with iSCSI, it fails, the disk disappears, and I am unable to restore the VM from a backup?

WARNING: Device mismatch detected for vgs_pve/lvmthin_data_tdata which is accessing /dev/sda instead of /dev/mapper/mpathb. 2025-08-06 08:27:45 WARNING: Thin pool vgs_pve/lvmthin_data has unexpected transaction id 3, expecting 9. 2025-08-06 08:27:45

device-mapper: message ioctl on (252:8) failed: Device or resource busy Failed to process message "delete 3".

sounds like there are low-level issues with the storage. Please open a new thread pinging @fiona and providing the more details about the storage configuration. Proxmox VE allows directly configuring LVM on top of iSCSI, but not LVM-thin, I suppose you set that up manually?

I have tried fix it but I keep getting the same error or a sliently different one and still no luck

What steps did you take? Please also share the full system journal from the current boot in the new thread.

 

[fiona]

Hi,
for all the people who had issues with their LVM thin pools and needed to repair them ( @Jedis @Kevo @Damon1974 @thomascgh ): Could you provide the /var/log/apt/term.log from during the update (please check if it was already rotated, then the relevant logs might be in term.log.1.gz etc. instead) as well as the system journal from the time right before the upgrade and until the pool was repaired, e.g. the command could be journalctl --since "2025-08-05 17:00:00" --until "2025-08-07 11:00:00" > /tmp/journal.txt with adapted time stamps.

 

[fiona]

Hi,

Congrats to the team on another huge milestone achieved!
I've seen there were plans to incorporate the ProxLB module into Proxmox VE v9.0. Has that been revised?

AFAIK, there are no plans to incorporate ProxLB, that is a completly separate project. The roadmap entry mentioned there is for a native solution within Proxmox VE. For the Proxmox VE 9 release, the focus for HA was on affinity rules. But the release also includes changes to the RRD data, making pressure stall information readily available which is an important building block required for resource scheduling/load balancing, which will be part of a future release.

 

[fiona]

Hi,

Hello, I don't know if this is right place to report but I have issues with running OPNSense in KVM with 6.14 kernel. I had this problem even with PVE 8 with 6.14 kernel. When I switch back to kernel 6.8 everything works. I don't know how to debug it because there is not error in journalctl in host. OPNSense inside KVM starting to boot but then it says "bus error" and just cannot mount storage and continue to boot. I tried switch to UEFI, to Q35, tried all possible controllers and IDE/SATA/VirtIO/SCSI options to connect hard disk but neither is working with 6.14. Anyone having same issue?

PVE host CPU is: 4 x Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (1 Socket)

Edit: Even OPNSense install ISO cannot boot.

please open a separate thread and ping @fiona there, providing the VM configuration qm config <ID> as well as the output of pveversion -v. Do you have the latest BIOS updates/CPU microcode installed? https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_firmware_cpu

P.S. Another place where this could've been posted is the kernel 6.14 thread: https://forum.proxmox.com/threads/o...e-8-available-on-test-no-subscription.164497/ but a dedicated thread is less noise and is easier to follow along

 

[tithitoy]

It's interesting that the memory usage is 101.15% on the pfSense (2.8.0) VM in PVE 9.
On PVE 8, the memory usage was around 70-75%. That was also incorrect, but not over 100%.

The memory balloon is off due to a missing implementation in FreeBSD to calculate the memory usage in QEMU.
Source: https://github.com/aborche/qemu-guest-agent/issues/19

View attachment 88892
View attachment 88894

monitor:

# info memdev
memory backend: pc.ram
  size:  4294967296
  merge: true
  dump: true
  prealloc: false
  share: false
  reserve: true
  policy: default
  host nodes:

# info memory_size_summary
base memory: 4294967296
plugged memory: 0

Memory usage in pfSense:
View attachment 88895

I have same problem.



Memory usage in pfsense:

 

[down16]

Me too (QUEMU Guest Agent installed)

 

[aaron]

@tithitoy @down16
This is not a problem, but due to the BSDs not reporting back any details regarding their internal memory usage. Therefore, we can only fall back to the host memory view, which has now changed a bit as we account for the overall overhead better.
In the past, the used memory shown in the Summary panel also never matched what you would see reported by pfsense/OPNsense.

You can compare the reported memory usage infos if you go the "Monitor" panel of a VM and type in info balloon. Compare the output from a FreeBSD based VM (e.g. OPNsense/pfsense) to one that runs Linux or Windows (with the Ballooning Agent running).

 

[paulsk]

Fair warning - I have an optical NIC in my node that uses the bnxt_en driver (lspci says it is a Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller). After the upgrade my network interfaces were renamed from enfp2xxx to enfp1xxx and broke networking. So be ready for that if you have similar hardware. Not a hard thing to fix, though.

Not pointing the finger at Proxmox though, maybe a Debian thing?

 

[fiona]

Hi,

Fair warning - I have an optical NIC in my node that uses the bnxt_en driver (lspci says it is a Broadcom Inc. and subsidiaries BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller). After the upgrade my network interfaces were renamed from enfp2xxx to enfp1xxx and broke networking. So be ready for that if you have similar hardware. Not a hard thing to fix, though.

Not pointing the finger at Proxmox though, maybe a Debian thing?

please see: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_naming_conventions for how to pin network interface names and avoid such issues in the future. Note that this is also mentioned in the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Network

 

[paulsk]

Hi,

please see: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_naming_conventions for how to pin network interface names and avoid such issues in the future. Note that this is also mentioned in the upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Network

Hello, thank you very much for the tips and advice!

I feel a bit humbled, as I did read through the guide, but it did not register to me as something to look out for, as the default naming convention had not ever failed me previously and I thought it would be of more relevance to the people using ethX device names.

Thanks again!

 

[ZeratuLx]

I booted into Linux 6.8.12-13-pve, and everything is working. On 9.0.3 (on the 6.14.8-2-pve-signed kernel). caught VM 100 qmp command 'query-proxmox-suppor' error when trying to start windows server, but booted from the old Linux 6.8.12-13-pve kernel and everything works. On that machine, PCI 3 devices are bridged (2 network controllers and a virtual mellanox network card, I don't blame mellanox, it works on 3 Linux machines)

Asrock B550 Pro4
Ryzen 5 4750 GE Pro
64 Gb DDR4 ECC

proxmox-ve: 9.0.0 (running kernel: 6.8.12-13-pve)
pve-manager: 9.0.3 (running version: 9.0.3/025864202ebb6109)
proxmox-kernel-helper: 9.0.3
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14: 6.14.8-2
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
amd64-microcode: 3.20250311.1
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.6
libpve-rs-perl: 0.10.7
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2
lxc-pve: 6.0.4-2
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.11-1
proxmox-backup-file-restore: 4.0.11-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.1
proxmox-kernel-helper: 9.0.3
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.0
proxmox-widget-toolkit: 5.0.5
pve-cluster: 9.0.6
pve-container: 6.0.9
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.16-3
pve-ha-manager: 5.0.4
pve-i18n: 3.5.2
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.16
smartmontools: 7.4-pve1
spiceterm: 3.4.0
swtpm: 0.8.0+pve2
vncterm: 1.9.0
zfsutils-linux: 2.3.3-pve1


affinity: 0,2,4,6,8,10
agent: 1
bios: ovmf
boot: order=virtio0
cores: 6
cpu: host,flags=+ibpb;+virt-ssbd;+amd-ssbd;+hv-tlbflush
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:04:00,pcie=1
hostpci1: 0000:05:00,pcie=1
hostpci2: 0000:01:02.2,pcie=1
hotplug: network,usb
machine: pc-q35-9.2+pve1
memory: 18432
meta: creation-qemu=8.1.5,ctime=1712381885
name: WinServer2025
net0: virtio=BC:24:**:**:**:**,bridge=vmbr0,link_down=1,mtu=1
net1: virtio=BC:24:**:**:**:**,bridge=vmbrvl,mtu=1
numa: 0
ostype: win11
protection: 0
scsihw: virtio-scsi-single
smbios1: uuid=28c1859d-2074-****-****-d0e2a8082e63,manufacturer=QXNyb2Nr,product=QjU1MCBQcm80,base64=1
sockets: 1
startup: order=2,up=40,down=200
tpmstate0: local-lvm:vm-100-disk-1,size=4M,version=v2.0
vga: virtio-gl
virtio0: local-lvm:vm-100-disk-6,aio=native,cache=none,discard=on,iothread=1,size=468G
virtio1: agi1781tb:100/vm-100-disk-0.qcow2,aio=native,cache=none,discard=on,iothread=1,size=120G
vmgenid: 8b4828fd-05c2-****-****-42f3c8b4fe55

I also noticed that even after stopping the frozen machine, the RAM is not released.

 

[Mr.Goodcat]

The update to v9 worked flawlessly.

However, since then dmesg/journalctl spams "i2c i2c-3: sendbytes: NAK bailout" every 10s.
The system uses a Ryzen 5600G on an Asrock X470D4U board with the latest BIOS and BMC firmware installed.
Checking /sys/bus/i2c/devices/i2c-3/name yields AST DDC bus. So it seems there is a new issue with the Aspeed BMC/IPMI chip. The error persists across (warm and cold) reboots.

Samba shares are also acting up, i.e. keep freezing when being accessed by windows clients.

So far there's nothing in logs that helps nailing down the causes. Any ideas?

 

[empirical6355]

I'm having an interesting issue related to PCI passthrough after I did the upgrade to 9.0. The one VM that I'm using with PCI passthrough will not boot, and seems to fail before even getting to the BIOS. I try to open the console to see what's going on and see pvestatd[1633]: VM 103 qmp command failed - VM 103 qmp command 'query-proxmox-support' failed - unable to connect to VM 103 qmp socket - timeout after 51 retries

The Proxmox UI will say it's running, but upon trying to shut it down the process will seemingly just hang. It shows as stopped in the UI but htop will show the VMID still has a process associated, in my case consuming a lot of CPU


Intel N97 in an Odroid H4+, trying to pass through the onboard ASMedia ASM1064 SATA controller

Wondering if anyone else is having this sort of issue. Happy to provide more logs or details but I am not sure what is relevant.

I followed the proxmox guide on doing PCI passthrough, https://pve.proxmox.com/wiki/PCI(e)_Passthrough

lspci -nnk

03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1064 Serial ATA Controller [1b21:1064] (rev 02)
        Subsystem: ASMedia Technology Inc. Device [1b21:2116]
        Kernel driver in use: vfio-pci
        Kernel modules: ahci
        
        
cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt consoleblank=120

root@odroid:~# lsmod | grep vfio
vfio_pci               16384  1
vfio_pci_core          86016  1 vfio_pci
vfio_iommu_type1       49152  1
vfio                   65536  8 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd               110592  1 vfio
irqbypass              12288  3 vfio_pci_core,kvm

root@odroid:~# dmesg | grep -e DMAR -e IOMMU
[    0.029233] ACPI: DMAR 0x00000000733EB000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.029263] ACPI: Reserving DMAR table memory at [mem 0x733eb000-0x733eb087]
[    0.071538] DMAR: IOMMU enabled
[    0.177374] DMAR: Host address width 39
[    0.177376] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.177387] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.177392] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.177398] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.177402] DMAR: RMRR base: 0x0000007c000000 end: 0x000000803fffff
[    0.177406] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.177408] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.177411] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.182530] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.446607] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.668793] DMAR: No ATSR found
[    0.668795] DMAR: No SATC found
[    0.668797] DMAR: dmar0: Using Queued invalidation
[    0.668802] DMAR: dmar1: Using Queued invalidation
[    0.669591] DMAR: Intel(R) Virtualization Technology for Directed I/O

 

[fiona]

Hi,
@ZeratuLx @empirical6355 please share the excerpt from the system journal around the time you start the VM. Is there anything in the VM <ID> - Start task log? If possible, can you try if booting without passthrough works?

Please run apt install pve-qemu-kvm-dbgsym gdb and then, when the VM is in the stuck state,
gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/123.pid) with the correct ID instead of 123 to obtain a backtrace. What does the subtree in ps faxl for the VM look like when it's in the stuck state?

Do you both have the latest BIOS updates/CPU microcode installed? https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_firmware_cpu

@empirical6355 please share the full VM config with qm config 103. EDIT: likely not related, but in general, using an odd CPU count like 3 for VMs is not recommended, as that doesn't appear much in the wild either

@ZeratuLx

I also noticed that even after stopping the frozen machine, the RAM is not released.

Is there still a process associated with the VM at this point in the output of ps aux? What does systemctl status 123.scope with the correct ID instead of 123 show?

 

[Stoiko Ivanov]

My question is : should I update the packages before running “apt dist-upgrade” ? (I don't think so, but I'd rather ask for your opinion)

see https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Prerequisites:

Upgraded to the latest version of Proxmox VE 8.4 on all nodes.
Ensure your node(s) have correct package repository configuration (web UI, Node -> Repositories) if your pve-manager version isn't at least 8.4.1.

(that is before changing the sources to trixie - you should install all pending updates)
I hope this helps!

 

[empirical6355]

Hi,
@ZeratuLx @empirical6355 please share the excerpt from the system journal around the time you start the VM. Is there anything in the VM <ID> - Start task log? If possible, can you try if booting without passthrough works?

Please run apt install pve-qemu-kvm-dbgsym gdb and then, when the VM is in the stuck state,
gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/123.pid) with the correct ID instead of 123 to obtain a backtrace. What does the subtree in ps faxl for the VM look like when it's in the stuck state?

Do you both have the latest BIOS updates/CPU microcode installed? https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_firmware_cpu

@fiona when you say "booting without passthrough" do you mean changing the kernel cmdline and rebooting, or just starting the VM without the PCI passthrough? I tried the latter, removing the PCI passthrough, and it works without issue. Boots up in seconds, console works as expected.

Thanks for the quick response! Yes, latest BIOS and microcode is installed for my system

Attached is the output of the System Log from the time I started the VM to the time I'm writing this, multiple minutes with the VM stuck

There's nothing in the VM Start task log, just TASK OK

The ps faxl command yields a single line related to this VM, no subtree, and it's a very long line: 7 0 45790 1 20 0 11122924 10549288 ptrace tLl ? 4:40 /usr/bin/kvm -id 103 -name unraid,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/103.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/103.pid -daemonize -smbios type=1,uuid=a0f2a67e-fc2e-47be-af10-8651ae4b65ea -object {"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"} -blockdev {"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.fd"},"node-name":"pflash0","read-only":true} -blockdev {"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/zvol/rpool/data/vm-103-disk-0","node-name":"e454aa0843523a51981a6f57fddc1ac","read-only":false},"node-name":"f454aa0843523a51981a6f57fddc1ac","read-only":false,"size":540672},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"} -smp 3,sockets=1,cores=3,maxcpus=3 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/103.vnc,password=on -cpu host,+kvm_pv_eoi,+kvm_pv_unhalt -m 10240 -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5 -device vmgenid,guid=7a3a0d32-b025-496e-b394-8cc0620cc52a -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device qemu-xhci,p2=15,p3=15,id=xhci,bus=pci.1,addr=0x1b -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device vfio-pci,host=0000:03:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device usb-host,bus=xhci.0,port=1,vendorid=0x0781,productid=0x5506,id=usb0,bootindex=100 -device VGA,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/103.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:78cea5efcb40 -netdev type=tap,id=net0,ifname=tap103i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=BC:24:11:3A:45:A5,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256 -machine pflash0=pflash0,pflash1=drive-efidisk0,type=pc+pve0

Interestingly, the GDB command hangs for over 60 seconds and I'm unable to cancel it. See photo. Eventually, maybe after 5-10 minutes it returned, I'll attach the log

Here is the output of qm config 103

agent: 1
balloon: 4096
bios: ovmf
boot: order=usb0
cores: 3
cpu: host
description: - Unraid VM with PCI SATA and USB Passthrough
efidisk0: local-zfs:vm-103-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00.0
memory: 10240
meta: creation-qemu=9.0.2,ctime=1735932528
name: unraid
net0: virtio=BC:24:11:3A:45:A5,bridge=vmbr0,firewall=1,tag=100
numa: 0
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=a0f2a67e-fc2e-47be-af10-8651ae4b65ea
sockets: 1
startup: order=2,up=300,down=300
tags: nas
usb0: host=0781:5506
vmgenid: 7a3a0d32-b025-496e-b394-8cc0620cc52a

I will update to 4 CPUs, thank you for that tip!

 

[SteveITS]

how to pin network interface names

Any chance on adding

pve-network-interface-pinning

to PVE 8? In a vacuum I think I'd prefer to set that in a controlled/known environment rather than upgrade, lose connectivity, fix it (presumably), and then change interface names again (from new to newer).

 

[fiona]

@fiona when you say "booting without passthrough" do you mean changing the kernel cmdline and rebooting, or just starting the VM without the PCI passthrough? I tried the latter, removing the PCI passthrough, and it works without issue. Boots up in seconds, console works as expected.

Yes, that is what I meant. Thanks for the feedback! We'll need to investigate kernel changes in this area, as it very much sounds like a kernel issue from the feedback we got until now. A workaround should be booting an older kernel in the meantime.