High load on opnsense vm after upgrade to pve 9

[aklausing]

HI,

I just updated my environment to pve 9
- 2 node cluster with no shared storage
- both host use the same hardware (Ryzen 5 5600G, same mb, same Memory (64Gib))
- some vms get replicated for HA

Now I have some weird issues with my virtual opnsense vm (FreeBSD) that was running fine before the upgrade. Wehen I start the VM on one node the memory usage goes to over 99% after a very short time and stays there. The management interface of the firewall reports around 23% usage for memory.





top inside the opnsense vm also reports around 23-25% mem usage.

When I migrate the vm with HA to the other node also the CPU raises to over 100% (the peak was 126%) but only on the pve side. Inside the vm the cpu usage seems absolutely normal but all KVM processes on the host consume a lot of cpu.

Updating the opnsense to 25.7 did not resolve the problem

This is the vm config:

agent: 1
balloon: 0
boot: order=sata0;sata1;net0
cores: 2
cpu: x86-64-v3
cpuunits: 200
machine: q35
memory: 8192
meta: creation-qemu=8.0.2,ctime=1690449616
name: vaultdoor
net0: virtio=22:E6:9F:,bridge=vmbr0
net1: virtio=9E:8E:A1,bridge=vmbr101
net2: virtio=66:C4:B9:,bridge=vmbr102
net3: virtio=56:38:97,bridge=vmbr0,tag=234
net4: virtio=7A:1F:12:,bridge=vmbr0,tag=122
numa: 0
onboot: 1
ostype: other
parent: Update_25_7
protection: 1
sata0: zfs-nvme:vm-666-disk-0,discard=on,size=100G
sata1: none,media=cdrom
scsihw: virtio-scsi-single
smbios1: uuid=64af0c63-
startup: order=1
tags: dns;firewall;network;nvme
vmgenid: 8560fd4a-

[Update_25_7]
agent: 1
balloon: 0
boot: order=sata0;sata1;net0
cores: 2
cpu: x86-64-v3
cpuunits: 200
machine: q35
memory: 8192
meta: creation-qemu=8.0.2,ctime=1690449616
name: vaultdoor
net0: virtio=22:E6:9F:,bridge=vmbr0
net1: virtio=9E:8E:A1:,bridge=vmbr101
net2: virtio=66:C4:B9:,bridge=vmbr102
net3: virtio=56:38:97:,bridge=vmbr0,tag=234
net4: virtio=7A:1F:12:,bridge=vmbr0,tag=122
numa: 0
onboot: 1
ostype: other
protection: 1
sata0: zfs-nvme:vm-666-disk-0,discard=on,size=100G
sata1: none,media=cdrom
scsihw: virtio-scsi-single
smbios1: uuid=64af0c63-
snaptime: 1754636475
sockets: 1
startup: order=1
tags: dns;firewall;network;nvme
vmgenid: 8560fd4a-2858-

This is the only vm showing this behaviour after the upgrade. All other vms run as usual without any issues.

Any suggestions?

Thanks in advance
Andreas

 

[sterzy]

Yes the memory accounting logic here has changed. This is more or less expected, see [1].

[1]: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#VM_Memory_Consumption_Shown_is_Higher

 

[aklausing]

Yes the memory accounting logic here has changed. This is more or less expected, see [1].

[1]: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#VM_Memory_Consumption_Shown_is_Higher

One should read behind the "I don't use GPU passthtough" section of the 8to9 doc

Any suggestions on the high CPU load after vm migration?

 

[IceOfWraith]

I'm dealing with a similar issue, but I did a clean install of 9 rather than an upgrade. The memory reporting is confusing, but doesn't seem to be an issue. The CPU becomes an issue because the VM stops responding eventually. I haven't been able to track down the root cause yet, but when it happens the CPU is running at 70-80% utilization and I can't access anything on the network anymore. I'm unable to get to the OPNSense management interface and all traffic comes to a standstill.

If you have any ideas how to diagnose, I'm all ears. The OPNSense logs don't show any issues after a reboot so I'm not sure where else to start digging.

 

[H4R0]

I'm also running multiple opnsense instances on Proxmox and did not notice any CPU increase since PVE9

@aklausing you should install `os-qemu-guest-agent` plugin in opnsense and enable `Qemu Guest Agent` in Proxmox VM Option s. Then you get the real memory usage on Proxmox.

 

[Fredouil]

Hello, I too am experiencing the same thing. My RAM is always at 100% on the proxmox side and when I look on my opnsense machine, it is only at 30% exactly the same problems, I also specify that my CPU is in Hosts and that I do not use ballooning for the RAM. The qemu agent is installed and functional. On pve 8 I did not have this problem I am in enterprise version on pve and Opnsense

 

[KingDigweed]

I've upgraded to PVE9 and have no performance issues, just incorrect reporting of RAM usage at 100% with my OPNsense VM. I'm aware of the advice provided in the PVE8 --> PVE9 documentation, but I wanted to check if the QEMU guest agent is supposed to resolve the reporting issue and help Proxmox display more accurate RAM usage?

I've always had the guest agent installed and running as the plugin described below, and I can see the difference with it running vs stopped (such as the guest's IP info), but both memory usage and host memory usage of the VM in the Proxmox web GUI is still displayed at 100%.

@H4R0 is your OPNsense VM actually reporting accurate RAM usage in the Proxmox webgui? If so, something must be broken with my setup, although I have tried re-installing the plugin to no avail. Ballooning is enabled if that makes a difference here.

I'm also running multiple opnsense instances on Proxmox and did not notice any CPU increase since PVE9

@aklausing you should install `os-qemu-guest-agent` plugin in opnsense and enable `Qemu Guest Agent` in Proxmox VM Option s. Then you get the real memory usage on Proxmox.

 

[andre.g]

I have recently upgraded Several existing Clusters to Proxmox 9, and also made complete fresh install on new nodes with Proxmox 9.03 and latest OPNsense 25.7.6 with ZFS on 3 virtual disk...

The issue, people here, on OPNsense Forum and Linus Tech Tips moaning about, is an hard fact and should be addressed. I even, which I never did on other production datacenter nodes, enabled for this new AMZ Ryzen Node, the Quemu-Agent, which is officially supported until FreeBSD 12 Release (we have 14 on FreeBSD). I also enabled it after on the Proxmox UI itself and checked on the OPNsense Node if its running. The issue persist. Also the CPU Values are wrong in Proxmox. We have a mixed environment, Data-Center on Subs, Home Labs not. We Monitor with NetData and CheckMK. Receiving wrong/false Positives you can not rely on is not only anyoing, can lead to ignorance and real issues...

So, dear Proxmox Team: Please address this issue! Thanks @ProxmoxSecurityAdvisory

 

[KingDigweed]

It does seem strange (and is a shame) that the advice is just "this is expected". Are we really being told that our RAM usage stats are now just broken until further notice?

It would be fine if this was just a way of saying "this was in the patch notes, you'll have to do [xyz] to get it working again", yet it seems that many of our VMs' memory stats have just become useless and will remain that way.

So yes, is there anything that can be done (on either side) to restore functionality? Is this not a regression?

 

[andre.g]

It is also funny, that NetData Agents reads different values out of the kvm...in this example we have 3 ZFS Disk each 64GB. The Netdata Agent installed as official Plugin from NetData and its Community, reads out an available Disk Size of 1TB (Proxmox NVME) and lists each of the Disk with 106 GB Volume.
Not sure, if its due Debian new Kernel Features, on Proxmox itself, but this issue is only Release 9. On 8.4.6 deployed OPNsense Firewall, the things are more or less correct.

 

[H4R0]

@H4R0 is your OPNsense VM actually reporting accurate RAM usage in the Proxmox webgui? If so, something must be broken with my setup, although I have tried re-installing the plugin to no avail. Ballooning is enabled if that makes a difference here.

I don't have ballooning enabled since I had problems with that during memory pressure.

Proxmox reports 2.5G, while the OPNsense GUI reports 2.4G, so very close.

If I disable the qemu guest agent the Proxmox report is completely wrong though.

Tried to do some more analysis on the 100mb difference but man FreeBSD is crap, there isn't even `free` available