[Proxmox VE 9.0 BETA] Slow VM UEFI post since upgrading

[Bent]

As suggested by @fweber (@fiona), I’m starting a new thread to track this issue.

We’ve encountered a significant problem in PVE 9: VM boot times (using OVMF) have become extremely slow. It now takes roughly 1 minute and 30 seconds from hitting “Start” to seeing any console output or UEFI POST.

This issue wasn’t present in PVE 8.x — VM startup there was almost instantaneous by comparison.

During this delay, CPU usage is consistently high (around 75–80%, regardless of how many cores are assigned), and RAM usage hovers at about 100 MB.

Below are two screenshots showing the VM state during this stall:




And here's the config of one of the affected VMs — though we’ve confirmed the issue occurs on all VMs across all nodes in the cluster:

agent: 1
bios: ovmf
boot: order=virtio0
cores: 4
cpu: x86-64-v3
efidisk0: ceph-nvme01:vm-117-disk-1,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network
machine: q35
memory: 4096
name: <redacted>
net0: virtio=F2:AC:38:7B:9A:A6,bridge=vmbr0,tag=100
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-single
smbios1: uuid=8b8a56a6-0684-4b2b-a50a-69b23b789235
sockets: 1
tablet: 0
tags: <redacted>
vga: virtio
virtio0: ceph-nvme01:vm-117-disk-0,discard=on,iothread=1,size=20G
vmgenid: 3890b812-a1fc-4331-a4c0-49d88c7632a6

Some more information:

root@node02 ~ # pveversion
pve-manager/9.0.0~11/c474e5a0b4bd391d (running kernel: 6.14.8-2-pve)
root@node02 ~ # qemu-system-x86_64 --version
QEMU emulator version 10.0.2 (pve-qemu-kvm_10.0.2-4)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers

Hi,

it is a known issue without using the proper cache mode, but Proxmox VE 9 should set the proper cache mode for the EFI disk: https://git.proxmox.com/?p=pve-stor...3cb0c3398c9fc19d305d9c36a74a4797715d009e#l564


What do you get when you query the cache mode for the image

rbd config image get vm-117-disk-1 rbd_cache_policy

?

Is this a PVE-managed or an external cluster? Do you maybe override the rbd_cache_policy in your Ceph client configuration or somewhere?

Here's the output I get:

root@node02 ~ # rbd --pool ceph-nvme01 config image get vm-117-disk-1 rbd_cache_policy
rbd: rbd_cache_policy is not set

I also checked the Ceph client configuration:

root@node02 ~ # grep rbd_cache_policy /etc/ceph/ceph.conf /etc/pve/ceph.conf
root@node02 ~ #

So there's no global override set either.

Just to clarify: this Ceph cluster was created and is fully managed by Proxmox VE (PVE-managed), no external cluster or manual Ceph deployment.

Could using KRBD be the issue?



Would you recommend manually setting the rbd_cache_policy to writeback for the EFI disk, or is there a more appropriate fix in PVE 9 for this scenario?

 

[fiona]

Hi,
as suggested by Friedrich in the other thread:

• check whether setting the machine version to 9.2+pve1 (Hardware->Machine, check the Advanced Settings) helps and include your observations in the new thread?

There also is a proposed patch that should fix the issue even with newer machine type: https://lore.proxmox.com/pve-devel/20250728084155.14151-1-f.ebner@proxmox.com/T/#u

Testing with older machine type would still be appreciated too, to verify it's the same issue.

 

[Bent]

Hi,
as suggested by Friedrich in the other thread:

• check whether setting the machine version to 9.2+pve1 (Hardware->Machine, check the Advanced Settings) helps and include your observations in the new thread?

There also is a proposed patch that should fix the issue even with newer machine type: https://lore.proxmox.com/pve-devel/20250728084155.14151-1-f.ebner@proxmox.com/T/#u

Testing with older machine type would still be appreciated too, to verify it's the same issue.

Is there an easy way to apply these patches without manually rebuilding the affected packages?
Alternatively, do you know when they’ll be included in the official repositories?

If you can provide a command to apply them locally, I’d be happy to test it!

 

[fiona]

Is there an easy way to apply these patches without manually rebuilding the affected packages?

The only other way is to patch the relevant Perl file and run systemctl reload-or-restart pvedaemon.service pveproxy.service pvescheduler.service pvestatd.service.

Alternatively, do you know when they’ll be included in the official repositories?

The patch has been applied to git already, so it will be included in the next package version, which probably won't take more than a few days during the beta now.

 

[Bent]

The only other way is to patch the relevant Perl file and run systemctl reload-or-restart pvedaemon.service pveproxy.service pvescheduler.service pvestatd.service.

The patch has been applied to git already, so it will be included in the next package version, which probably won't take more than a few days during the beta now.

I found the first file:

src/PVE/QemuServer/OVMF.pm -> /usr/share/perl5/PVE/QemuServer/OVMF.pm

Where can I find these in the system?

src/test/cfg2cmd/efi-ovmf-without-efidisk.conf.cmd
src/test/cfg2cmd/efi-raw-template.conf.cmd
src/test/cfg2cmd/efi-raw.conf.cmd
src/test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd
src/test/cfg2cmd/efi-secboot-and-tpm.conf.cmd
src/test/cfg2cmd/q35-linux-hostpci-mapping.conf.cmd
src/test/cfg2cmd/q35-linux-hostpci-multifunction.conf.cmd
src/test/cfg2cmd/q35-linux-hostpci-template.conf.cmd
src/test/cfg2cmd/q35-linux-hostpci-x-pci-overrides.conf.cmd
src/test/cfg2cmd/q35-linux-hostpci.conf.cmd
src/test/cfg2cmd/q35-simple.conf.cmd
src/test/cfg2cmd/sev-es.conf.cmd
src/test/cfg2cmd/sev-std.conf.cmd

 

[fiona]

The test are only executed when building the package, not installed on the system, so there is no need to apply that part of the patch

 

[Bent]

Thank you very much, the patch fixes the slow post times in my test!

Successfully tested with machine version 9.2+pve1 and machine version 8.2 on a Windows and Linux guest.

cat > OVMF_pm.patch <<'EOF'
diff --git a/src/PVE/QemuServer/OVMF.pm b/src/PVE/QemuServer/OVMF.pm
index 9789c2fb..8948651c 100644
--- a/src/PVE/QemuServer/OVMF.pm
+++ b/src/PVE/QemuServer/OVMF.pm
@@ -182,6 +182,10 @@ my sub generate_ovmf_blockdev {
         $format = 'raw';
     }
 
+    # Prior to -blockdev, QEMU's default 'writeback' cache mode was used for EFI disks, rather than
+    # the Proxmox VE default 'none'. Use that for -blockdev too, to avoid bug #3329.
+    $drive->{cache} = 'writeback' if !$drive->{cache};
+
     my $extra_blockdev_options = {};
     # extra protection for templates, but SATA and IDE don't support it..
     $extra_blockdev_options->{'read-only'} = 1 if drive_is_read_only($conf, $drive);

EOF

patch /usr/share/perl5/PVE/QemuServer/OVMF.pm < OVMF_pm.patch

 

[fiona]

Thank you for the reports and for testing, much appreciated!