pfSense VM keeps freezing/crashing

T

thimplicity

Member
Feb 4, 2022
73
9
13
44
Hi everyone,
I have installed proxmox on a box with
  • Intel N5105 4-core
  • 4*2.5GB I225 B3
  • 8GB RAM
  • 500GB NVMe
pfSense is the only VM running. I installed it based on instructions on the netgate website. WHEN it runs, it is fine, but once a day or so, it crashes or freezes. Most of the time it just reboots, today was a freeze. The syslog error message from proxmox is:

proxmox syslog
Code: 
Jul 18 17:09:33 pve pvedaemon[1117]: <root@pam> successful auth for user 'root@pam'
Jul 18 17:17:01 pve CRON[175424]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 18 17:17:01 pve CRON[175425]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 18 17:17:01 pve CRON[175424]: pam_unix(cron:session): session closed for user root
Jul 18 17:31:57 pve chronyd[900]: Received KoD RATE from 138.68.201.49
Jul 18 17:44:52 pve chronyd[900]: Received KoD RATE from 206.82.16.3
Jul 18 17:48:16 pve chronyd[900]: Received KoD RATE from 50.116.6.123
Jul 18 17:55:17 pve kernel: perf: interrupt took too long (3930 > 3928), lowering kernel.perf_event_max_sample_rate to 50750
Jul 18 17:56:05 pve QEMU[1148]: KVM internal error. Suberror: 3
Jul 18 17:56:05 pve QEMU[1148]: extra data[0]: 0x0000000080000b0e
Jul 18 17:56:05 pve QEMU[1148]: extra data[1]: 0x0000000000000031
Jul 18 17:56:05 pve QEMU[1148]: extra data[2]: 0x0000000000000083
Jul 18 17:56:05 pve QEMU[1148]: extra data[3]: 0x00000008005ecfe0
Jul 18 17:56:05 pve QEMU[1148]: extra data[4]: 0x0000000000000001
Jul 18 17:56:05 pve QEMU[1148]: RAX=00000008005ecf40 RBX=fffffe0000f53070 RCX=00000000c0000101 RDX=00000000ffffffff
Jul 18 17:56:05 pve QEMU[1148]: RSI=0000000800bde160 RDI=fffffe0000f53070 RBP=fffffe0000f53060 RSP=fffffe0000f52f90
Jul 18 17:56:05 pve QEMU[1148]: R8 =a7b0f469b0f0e363 R9 =064248be4684b7ae R10=e400000000000000 R11=00000008005ecf40
Jul 18 17:56:05 pve QEMU[1148]: R12=0000000000000000 R13=00000000207dc000 R14=000000080128e400 R15=000000080128e750
Jul 18 17:56:05 pve QEMU[1148]: RIP=ffffffff8135a851 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
Jul 18 17:56:05 pve QEMU[1148]: ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Jul 18 17:56:05 pve QEMU[1148]: CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
Jul 18 17:56:05 pve QEMU[1148]: SS =0000 0000000000000000 ffffffff 00c00000
Jul 18 17:56:05 pve QEMU[1148]: DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Jul 18 17:56:05 pve QEMU[1148]: FS =0013 0000000800b520d0 ffffffff 00c0f300 DPL=3 DS   [-WA]
Jul 18 17:56:05 pve QEMU[1148]: GS =001b ffffffff84211000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Jul 18 17:56:05 pve QEMU[1148]: LDT=0000 0000000000000000 ffffffff 00c00000
Jul 18 17:56:05 pve QEMU[1148]: TR =0048 ffffffff83719808 00002068 00008b00 DPL=0 TSS64-busy
Jul 18 17:56:05 pve QEMU[1148]: GDT=     ffffffff83720008 00000067
Jul 18 17:56:05 pve QEMU[1148]: IDT=     ffffffff837187a0 00000fff
Jul 18 17:56:05 pve QEMU[1148]: CR0=80050033 CR2=ffffffff8135a851 CR3=00000008005ecf40 CR4=000006e0
Jul 18 17:56:05 pve QEMU[1148]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Jul 18 17:56:05 pve QEMU[1148]: DR6=00000000ffff0ff0 DR7=0000000000000400
Jul 18 17:56:05 pve QEMU[1148]: EFER=0000000000000d01
Jul 18 17:56:05 pve QEMU[1148]: Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??

pfSense syslog
Code: 
Jul 18 17:23:00    sshguard    81003    Exiting on signal.
Jul 18 17:23:00    sshguard    95856    Now monitoring attacks.
Jul 18 17:53:00    sshguard    95856    Exiting on signal.
Jul 18 17:53:00    sshguard    18377    Now monitoring attacks.
Jul 18 18:48:15    syslogd        kernel boot file is /boot/kernel/kernel
Jul 18 18:48:15    kernel        ---<<BOOT>>---

pveversion -v (see updated version in post below)
Code: 
proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

cat /etc/pve/qemu-server/101.conf
Code: 
#...
balloon: 0
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 2
cpu: kvm64
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/pfSense-CE-2.6.0-RELEASE-amd64.iso,media=cdrom,size=749476K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1657762558
name: pfSenseNEW
net0: virtio=06:EF:0A:B8:83:9D,bridge=vmbr2,firewall=1
net1: virtio=B2:13:5A:B0:3F:0E,bridge=vmbr3,firewall=1
numa: 0
onboot: 1
ostype: other
parent: pf20220716
scsihw: virtio-scsi-pci
smbios1: uuid=8f41b222-6a11-4f8b-94d5-ec04854fa027
sockets: 1
startup: order=1
vga: qxl
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: b0e0ac34-a51c-467b-824b-434dbbb7796c

[pf20220713]
#...
balloon: 0
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 2
cpu: kvm64,flags=+aes
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/pfSense-CE-2.6.0-RELEASE-amd64.iso,media=cdrom,size=749476K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1657762558
name: pfSenseNEW
net0: virtio=06:EF:0A:B8:83:9D,bridge=vmbr2,firewall=1
net1: virtio=B2:13:5A:B0:3F:0E,bridge=vmbr3,firewall=1
numa: 0
ostype: other
runningcpu: kvm64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.2+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=8f41b222-6a11-4f8b-94d5-ec04854fa027
snaptime: 1657768156
sockets: 1
vga: qxl
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: b0e0ac34-a51c-467b-824b-434dbbb7796c
vmstate: local-lvm:vm-101-state-pf20220713

[pf20220715]
#...
balloon: 0
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 2
cpu: kvm64,flags=+aes
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/pfSense-CE-2.6.0-RELEASE-amd64.iso,media=cdrom,size=749476K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1657762558
name: pfSenseNEW
net0: virtio=06:EF:0A:B8:83:9D,bridge=vmbr2,firewall=1
net1: virtio=B2:13:5A:B0:3F:0E,bridge=vmbr3,firewall=1
numa: 0
ostype: other
parent: pf20220713
runningcpu: kvm64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.2+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=8f41b222-6a11-4f8b-94d5-ec04854fa027
snaptime: 1657913252
sockets: 1
vga: qxl
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: b0e0ac34-a51c-467b-824b-434dbbb7796c
vmstate: local-lvm:vm-101-state-pf20220715

[pf20220716]
#...
balloon: 0
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 2
cpu: kvm64,flags=+aes
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/pfSense-CE-2.6.0-RELEASE-amd64.iso,media=cdrom,size=749476K
machine: q35
memory: 2048
meta: creation-qemu=6.2.0,ctime=1657762558
name: pfSenseNEW
net0: virtio=06:EF:0A:B8:83:9D,bridge=vmbr2,firewall=1
net1: virtio=B2:13:5A:B0:3F:0E,bridge=vmbr3,firewall=1
numa: 0
ostype: other
parent: pf20220715
runningcpu: kvm64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.2+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=8f41b222-6a11-4f8b-94d5-ec04854fa027
snaptime: 1657972090
sockets: 1
vga: qxl
virtio0: local-lvm:vm-101-disk-1,size=32G
vmgenid: b0e0ac34-a51c-467b-824b-434dbbb7796c
vmstate: local-lvm:vm-101-state-pf20220716

I googled the error and most discussions seem to focus around the CPU type used for the VM. Could it be that the CPU is relatively new and not supported by proxmox yet? Honestly, no idea. Proxmox itself runs great, but it does not help, if the main and only VM does not :).

Thanks for some ideas!
 
Last edited:
I run pfSense on a number of proxmox hosts and it runs very reliably on various hardware although none the same as yours.

However, I tend to run them as 'default' - i.e machine type=i440fx and Bios=SeaBIOS, have you tried this?
 
bobmc said:
I run pfSense on a number of proxmox hosts and it runs very reliably on various hardware although none the same as yours.

However, I tend to run them as 'default' - i.e machine type=i440fx and Bios=SeaBIOS, have you tried this?
Click to expand...
Do i need to reinstall the whole thing or can I change that "on the fly"?
 
Neobin said:
  • [Is the bios/UEFI up-to-date? Can't hurt to check.]
  • Your PVE is definitely not. Update and reboot it afterwards. (You need to add the pve-no-subscription repository [1] and disable the enterprise one, if you do not have an active subscription.)
  • Try CPU type host for the VM.

[1] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo
Click to expand...
Completed 2. and 3. Regarding BIOS I ordered the box from AliExpress. Will need to check how to update the BIOS.

New pveversion -v
Code: 
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.4-1
proxmox-backup-file-restore: 2.2.4-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
 
Last edited:
2 Days 01 Hour 39 Minutes 14 Seconds and no more freezes or reboots. Sounds like 2. and 3. helped - thanks everyone *knockonwood*!

One more question. If I use host, do I have to use that setting for all VMs and can I still overprovision CPU cores?
 
  • Like
Reactions: Neobin
Nice to hear. Hopefully it stays that way. :)

thimplicity said:
If I use host, do I have to use that setting for all VMs
Click to expand...

No. But host is preferred for best performance and all CPU feature flags for the VMs; unless you need the compatibility for e.g. migration between nodes with different hardware (especially CPU types).

thimplicity said:
and can I still overprovision CPU cores?
Click to expand...

Over all VMs: Yes.
For a single VM: No. (But that is ever the case, no matter what CPU type you use.)

Further reading:
https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_cpu
 
Unfortunately an update on this one:
- proxmox runs rock solid since 10 days
- the pfSense VM froze once after 5 days and rebooted itself after 4 days.

I also put a fan in front of the server and temperatures are all below 50 celcius.

Not sure what else to try - maybe someone has some ideas
 
ness1602 said:
Are you doing something on the machine around the freeze period, like backups or similar?
Click to expand...
Not that I am aware off. Most of the time it seems to crash when I am doing something, like configurations (nothing out of the ordinary which should cause a crash or reboot that has not been done 1000 times before).
 
thimplicity said:
WHEN it runs, it is fine, but once a day or so, it crashes or freezes. Most of the time it just reboots, today was a freeze. The syslog error message from proxmox is:
Click to expand...
It appears that there is an issue running VMs on the recently available Jasper Lake (Intel 11th Gen Celeron) processors including N5095, N5105, N6005.
This is not specifically an Proxmox problem as it is also effecting Unraid and other similar systems.
It appears that there may be a Kernel/KVM bug that is behind the issue on all kernels prior to 5.19.4. In general people are reporting VMs are stable with this release (although it has only been a few days).
It does not appear to matter what the VM is running (CentOS, pfsense, OpenWRT, etc).
There is active discussion of the issue in this Proxmox thread and also here on the STH Topton forum.
The Proxmox forum thread discusses how to upgrade to kernel 5.19.4.

thimplicity said:
Regarding BIOS I ordered the box from AliExpress. Will need to check how to update the BIOS.
Click to expand...
This is pretty straight forward assuming you can workout what system you bought.
Most are shipping with boards from ChangWang (CWWK). You need to identify if you have V1, V2 or V3/V4.
V4 is just the V3 with the i225 chips upgraded to the i226.
Pictures of the different boards are here with links to the BIOS pages.
Just build a bootable USB using Ventoy, download the relevant ISO, copy it onto the USB and you are good to go.
 
  • Like
Reactions: bibubo9 and Neobin
OP did this fix your issue when updating bios? I just installed promox+pfsense -- it was running for 2-3 days and random restarted. Pfsense VM in promox was still up but uptime on pfsense just restarted and internet went out for 15min.
 
Hi, this thread has helped me solve my problem. Here is my summary post on the settings that seemed to have delivered the breakthrough. pfSense uptime is now 71 Days 16 Hours 12 Minutes 51 Seconds
 
thimplicity said:
Hi, this thread has helped me solve my problem. Here is my summary post on the settings that seemed to have delivered the breakthrough. pfSense uptime is now 71 Days 16 Hours 12 Minutes 51 Seconds
Click to expand...
Thanks for this. I updated the PVE kernel and it has been going 2 days strong. If I see the pfsense rebooting again, I will try your method.
I noticed you didn't do pci passthrough option, are you able to fully saturate 2.5gb?
 
terrybads said:
Thanks for this. I updated the PVE kernel and it has been going 2 days strong. If I see the pfsense rebooting again, I will try your method.
I noticed you didn't do pci passthrough option, are you able to fully saturate 2.5gb?
Click to expand...
I purely use it as a router with a 400mbit connection, so I do not know :). 400mbit is fine. I also read that even with 1-2 Gbit, passthrough or virtualization should not make much of a difference. Have not put it to the test myself.
 
  • Like
Reactions: terrybads
I bought one of these mini PCs and had the same issue and after some digging it appears that's a bug on the energy savings/turbo of the open source firmware for Intel CPUs.

I'm on PVE 8.2.2 and after enabling the non-free-firmware repo and installing the intel-microcode package, the issue was fixed for me.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back