Freebsd/OPNsense does not boot with cpu:host on kernel 5.11

[EugenMayer]

Due to fixing nested virtualization issues I switch to the 5.11 kernel line using

apt-get install pve-kernel-5.11

after that my freebsd/opnsense (21.1.5-amd64) box (FreeBSD 12), which is using cpu:host will no longer boot due to a kernel panic.
Are there any hints or known issues on how to fix this?

Other VMs using cpu:host (Linux based) are working just fine

Other threads mentioning the same issue

• OPNsense does not boot

Host CPU: AMD Ryzen 9 5950X 16-Core Processor

 

[Stefan_R]

Are you using an AMD or Intel CPU on the host? I believe Intel had some changes related to how the 'vmx' flag for nested virt is displayed to the guest in kernel 5.11, that could potentially trip BSD up. You can try to disable nesting specifically for this VM, by creating a custom CPU model that has a 'flags -vmx' entry (see 'man cpu-models.conf'). Could you also post your full VM config (qm config <vmid>)?

 

[EugenMayer]

Updated CPU in the original post (its an AMD)

This is the non-bootable configuration due to cpu: host

boot: dcn
bootdisk: scsi0
cores: 2
ide2: none,media=cdrom
memory: 6000
name: gateway
net0: virtio=<redacted>,bridge=vmbr30
net1: virtio=<redacted>,bridge=vmbr0
net2: virtio=<redacted>,bridge=vmbr0
numa: 0
onboot: 1
ostype: other
protection: 1
scsi0: local:400/vm-400-disk-1.qcow2,size=10G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=<redacted>
sockets: 1
startup: order=1
vga: serial0
cpu: host

 

[Stefan_R]

I tried replicating your setup, running FreeBSD 12 and 13 on an AMD Ryzen 7 2700X, but everything works perfectly fine here... I'm also running a pfSense box on 5.11 on a different 2nd gen AMD Ryzen (forgot the exact model atm) and it works just fine, all with cpu model 'host' of course. Anything else special about the VM in question? Is there maybe a (kernel) update available in the guest?

You could try and see if it is indeed nested virtualization related by disabling it for the VM in question. A bit hacky, but try the following:

qm set <vmid> --args '-cpu host,-svm,+kvm_pv_eoi,+kvm_pv_unhalt'

Alternatively, disable nesting entirely in the kernel module.

 

[EugenMayer]

Thanks for the effort and information Stefan! I will try to replicate that in an office our and post the kernel exception here.

1. there are no open kernel updates
2. Running FreeBSD:12 on Opnsense 21.1.5 (latest) so I suppose it is the latest FreeBSD 12 kernel

Why would I need nested virtualization in that VM anyway? So using your hack could be practical, wouldn't it?

 

[Stefan_R]

Why would I need nested virtualization in that VM anyway? So using your hack could be practical, wouldn't it?

Sorry that might have been badly phrased: I'm not saying it's hacky to disable nesting for a single VM, but doing it via the 'args' option is a bit bad, since that overwrites anything you set as the CPU in the GUI. It'll work, but to do it properly, you should look into custom CPU models and disable the 'svm' flag that way (see man cpu-models.conf).

 

[Gektor]

Same issue on Proxmox 7 with FreeBSD 12 (pfSense 2.5.2)
Host CPU: AMD Ryzen 9 5950X 16-Core Processor

 

[Gektor]

qm set <vmid> --args '-cpu host,-svm,+kvm_pv_eoi,+kvm_pv_unhalt'

Did not help at all...

 

[Stefan_R]

Same issue on Proxmox 7 with FreeBSD 12 (pfSense 2.5.2)
Host CPU: AMD Ryzen 9 5950X 16-Core Processor

There's no way of knowing if it is the same issue, a kernel panic can have a huge number of reasons... In general, if you're not sure, open a new thread. In your case it seems to be a bad page-fault, so... bad RAM? bad installation disk? corrupted disk? just guessing though...

 

[Gektor]

There's no way of knowing if it is the same issue, a kernel panic can have a huge number of reasons... In general, if you're not sure, open a new thread. In your case it seems to be a bad page-fault, so... bad RAM? bad installation disk? corrupted disk? just guessing though...

On two 5950x servers same... And on Unraid forums same problems, seems that bug with FreeBSD and Linux qemu. Switching CPU to EPYC ROME type fixing that issue.

 

[psylou]

On two 5950x servers same... And on Unraid forums same problems, seems that bug with FreeBSD and Linux qemu. Switching CPU to EPYC ROME type fixing that issue.

Same thing for me with ryzen 5 5600X! Upgraded Proxmox from 6 to 7 and opnsense paniced on start. Only your solution to put processor type to epyc rome fixed it. But Im losing Performance with that right?

 

[janssensm]

But Im losing Performance with that right?

Such wealth problems
Running virtual opnsense too on old opteron 6262HE without issues. I'm sure the new release based on FreeBSD 13 will solve a lot of these issues.
But to be honest i'm still running q35 3.1 to be sure that pcie is able to passthrough, so we're in the same boat .
Kudos to opnsense devs.

 

[Stefan_R]

Only your solution to put processor type to epyc rome fixed it. But Im losing Performance with that right?

For an OPNsense? Most likely not. QEMU's EPYC model is very close to real hardware, and only some esoteric features (as well as nesting) gets disabled, nothing that would slow down a router AFAICT.

 

[rockmox]

I have the exact same issue with AMD Ryzen 9 5950X 16-Core Processor. Temporarily selecting EPYC Rome as well.

 

[psylou]

The problem seems to be fixed. I was able to change back to host CPU.