Proxmox 8.13 + OPNSense 23.7 crash

[smittyj]

Hi,

I am running the latest Proxmox and OPNSense on a Protectli VP4650 (latest AMI BIOS). Within a few hours of booting the machine will lockup and display is nothing but random lines -- the only fix is a power cycle. I've had Protectli replace 2 units, this is the 3rd so I am prepared to rule out hardware. I ran a memtest for 4 hours with 0 errors. I've tried two different SSDs and fresh installs on both. I got my last replacement unit and installed Proxmox on it 6 days ago and everything seemed stable so I went and installed OPNSense on it this afternoon, only to discover 2 hours later -- same issue, had to unplugged the power. I've looked in the logs and I see nothing unusually before the point of the reboot. I am open to any ideas. All the configurations are defaults, the only change is the use of VLANs for Proxmox management and the virtual machines.

At this point, I am just about out of things to try other than ESX8, I don't want to run OPNSense on bare metal.

 

[Taomyn]

What's the config of the VM for OPNsense? I've been running it as a VM for almost a month with no issues on a small firewall appliance (Intel Core I3 N305, 4 x Intel NICs). I moved from a bare metal box to this with little difficulties.

Here is mine with a couple of items redacted:

agent: 1
balloon: 0
bios: seabios
boot: order=scsi0
cores: 4
cpu: host
hostpci0: 0000:01:00,pcie=1
hostpci1: 0000:03:00,pcie=1
hostpci2: 0000:04:00,pcie=1
hotplug: disk,network,usb,cpu
machine: q35
memory: 8192
meta: creation-qemu=8.1.2,ctime=1701086589
name: BART
net0: virtio=BC:24:11:78:54:D5,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-zfs:vm-100-disk-1,discard=on,iothread=1,size=64G
scsihw: virtio-scsi-single
smbios1: uuid=xxxxxxxx
sockets: 1
startup: order=2
tags: linux;vm
vmgenid: 84xxxxxxxx

 

[sherminator]

to be clear: The OPNsense VM is crashing your PVE host? uh.
I would try some other VM guest OS, a Debian for example. And also some LXC testing, to find out what *doesn't* crash your host.
What kind of storage setup do you use?

Maybe you could post your OPNsense VM configuration?

 

[smittyj]

The Proxmox host is 100% locked up when the OPNSense VM is running. I've tried the defaults when creating the VM (below) and I've also followed the KB from Protectli (https://kb.protectli.com/kb/opnsense-on-proxmox-ve/) which has different options selected. I have a CWWK i3-n305 running just fine but my VP4650 from Protectli (i5-10210U) crashes with either OPNSense parameters. Last night I decided to try ESXi8 and still this AM -- running, that never would have been possible on Proxmox. I am certain it is some combination of HW and SW not happy.

boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
hostpci0: 0000:04:00.0
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=8.1.2,ctime=1703018239
name: OPNSense01
net0: virtio=BC:24:11:9D:4A:57,bridge=vmbr0,firewall=1,tag=6
net1: virtio=BC:24:11:EF:F5:FA,bridge=vmbr0,firewall=1,tag=70
net2: virtio=BC:24:11:9B:54:A0,bridge=vmbr0,firewall=1,tag=10
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=43f903f1-31cd-4f21-8437-514e9287380e
sockets: 1
vmgenid: 0500ed42-a9a7-42ec-b2de-d216451b6467

 

[Taomyn]

I would try some of the settings I have and see if they help e.g. cpu, numa, machine

I do recall that enabling ballooning will cause problems and prevents all the RAM being detected.

 

[sherminator]

Did you try another guest OS, just for science?
I vaguely remember some trouble related to low power Intel CPUs, no?

 

[RolandK]

what's the specs of your HOST, especially RAM ?

 

[Joris L.]

silly checklist

* does the Protectli VP4650 have virtualisation enabled and/or hyperthreading ?
* is the Qemu-guest-agent installed in opnsense ?
* are you bonding network interfaces ?
* how many cores did you assign to the opnsense vm ?
* how much memory did you assign to the opnsense vm ?