proxmox 5.0.21-5-pve pfSense VM won't boot
- Thread starter mimino
- Start date
I'm using UEFI. AMD RX-427BB board.
Here's the full config:
Here's the full config:
Code:
balloon: 0
bios: ovmf
bootdisk: virtio0
cores: 2
sockets: 1
cpu: host
efidisk0: local-lvm:vm-100-disk-1,size=128K
hostpci0: 01:00,pcie=1
ide2: none,media=cdrom
machine: pc-q35-3.1
memory: 4096
name: pfSense
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=00a6f85c-0ed3-4707-96c1-b0a4232036f2
virtio0: local-lvm:vm-100-disk-0,size=32G
vmgenid: 84f6e97c-f175-4155-916e-7d2d216b4a7e
Code:
root@dfi:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-3-pve)
pve-manager: 6.0-12 (running version: 6.0-12/0a603350)
pve-kernel-helper: 6.0-12
pve-kernel-5.0: 6.0-11
pve-kernel-4.15: 5.4-9
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-9-pve: 4.15.18-30
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-3
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-7
libpve-guest-common-perl: 3.0-2
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-10
pve-docs: 6.0-8
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-4
pve-ha-manager: 3.0-3
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-13
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2
Last edited:
I am having the same problem here. Did anyone find a fix?
Edit - found a thread on reddit and changed the CPU type for the vm to one of the opteron variants instead of "host" and it will not boot. Without encryption support of course, but it at least boots.
Edit - found a thread on reddit and changed the CPU type for the vm to one of the opteron variants instead of "host" and it will not boot. Without encryption support of course, but it at least boots.
Last edited:
Mind sharing this reddit thread?
In my case proxmox won't start any VM with CPU type set to anything other than "host", without turning off KVM hardware virtualization. So I doubt this solution would work for me
Mind sharing this reddit thread?
In my case proxmox won't start any VM with CPU type set to anything other than "host", without turning off KVM hardware virtualization. So I doubt this solution would work for me
That's rather odd.
https://www.reddit.com/r/homelab/comments/dz4ecy/pfsense_boot_issue_after_proxmox_update/
In the reddit post I linked to a netgate link where someone is having an issue with straight kvm so I think this is technically a kvm issue.
Thanks, might try later, but as I said the chances of booting with anything other than 'host' are close to zero.
Latest kernel broke some things, we just need to get somebody knowledgeable to figure out what that is. Hoping that someone from proxmox staff might be willing to help us.
- I have a T620+ (AMD GX-420CA SOC with Radeon(tm) HD Graphics CPU, not sure what exactly you need to know)
-
Code:
root@pvesmall:~# pveversion -v proxmox-ve: 6.0-2 (running kernel: 5.0.21-5-pve) pve-manager: 6.0-12 (running version: 6.0-12/0a603350) pve-kernel-helper: 6.0-12 pve-kernel-5.0: 6.0-11 pve-kernel-4.15: 5.4-8 pve-kernel-5.0.21-5-pve: 5.0.21-10 pve-kernel-5.0.21-2-pve: 5.0.21-7 pve-kernel-5.0.21-1-pve: 5.0.21-2 pve-kernel-4.15.18-20-pve: 4.15.18-46 pve-kernel-4.15.18-9-pve: 4.15.18-30 ceph-fuse: 12.2.11+dfsg1-2.1+b1 corosync: 3.0.2-pve4 criu: 3.11-3 glusterfs-client: 5.5-3 ksm-control-daemon: 1.3-1 libjs-extjs: 6.0.1-10 libknet1: 1.13-pve1 libpve-access-control: 6.0-3 libpve-apiclient-perl: 3.0-2 libpve-common-perl: 6.0-7 libpve-guest-common-perl: 3.0-2 libpve-http-server-perl: 3.0-3 libpve-storage-perl: 6.0-9 libqb0: 1.0.5-1 libspice-server1: 0.14.2-4~pve6+1 lvm2: 2.03.02-pve3 lxc-pve: 3.2.1-1 lxcfs: 3.0.3-pve60 novnc-pve: 1.1.0-1 proxmox-mini-journalreader: 1.1-1 proxmox-widget-toolkit: 2.0-8 pve-cluster: 6.0-7 pve-container: 3.0-10 pve-docs: 6.0-8 pve-edk2-firmware: 2.20190614-1 pve-firewall: 4.0-7 pve-firmware: 3.0-4 pve-ha-manager: 3.0-3 pve-i18n: 2.0-3 pve-qemu-kvm: 4.0.1-5 pve-xtermjs: 3.13.2-1 qemu-server: 6.0-13 smartmontools: 7.0-pve2 spiceterm: 3.1-1 vncterm: 1.6-1 zfsutils-linux: 0.8.2-pve2 root@pvesmall:~# -
Code:
root@pvesmall:~# qm config 300 boot: c bootdisk: ide0 cores: 4 cpu: Opteron_G3 ide0: pvespace:vm-300-disk-0,size=100G ide2: none,media=cdrom memory: 4096 name: pfroute.jay.home net0: virtio=EA:5D:76:54:73:36,bridge=vmbr0 net1: virtio=E2:28:55:CC:A2:40,bridge=vmbr1 numa: 0 onboot: 1 ostype: other scsihw: virtio-scsi-pci smbios1: uuid=d2c4b2bf-224b-4dde-8d93-9df1d31ae8b2 sockets: 1 startup: order=1 tablet: 0 vmgenid: dac8834e-fc61-47bd-abd3-891137d3df7d root@pvesmall:~#
Had the same problem today on my beginnings of a homelab... I had just put the no-subscription apt source in /apt/sources.list and ran updates/upgrades/dist-upgrade.
According to apt logs, Kernel 5.0.21-10 was installed as part of those updates.
After rebooting the host, Pfsense refused to boot. Changing to KVM instead of Host for CPU type allowed it to boot but breaks some desired functionality.
Decided to give Tom's suggestion a try here (google found this thread with a few related keywords pretty fast):
>apt install pve-kernel-5.3
Fixed! Thanks Tom!!!
Pfsense working again with "host" type CPU.
This is on an A10-7850K CPU. Seems to be an AMD related regression in that kernel update.
According to apt logs, Kernel 5.0.21-10 was installed as part of those updates.
After rebooting the host, Pfsense refused to boot. Changing to KVM instead of Host for CPU type allowed it to boot but breaks some desired functionality.
Decided to give Tom's suggestion a try here (google found this thread with a few related keywords pretty fast):
>apt install pve-kernel-5.3
Fixed! Thanks Tom!!!
Pfsense working again with "host" type CPU.
This is on an A10-7850K CPU. Seems to be an AMD related regression in that kernel update.
Last edited:
Why are you setting CPU Type to
host for pfSense VM?Based on my current knowledge,
host CPU Type is required (and meant) for Nested Virtualization Use Cases.I haven't tested any "non-virtualizing" VM to run with
host CPU Type, but I assume it negatively impacts the instruction set architecture (ISA) presented to VM by host. And this might be causing system hungs described.
I've experienced issues with system hungs of pfSense 2.4.4-RELEASE-p3 VMs on kernels from
5.0.21-3-pve up to 5.0.21-10-pve, so I have deployed 5.3.7-1-pve and it fixed my issue for the moment.When referring to pve-kernel-5.3, there are still there versions of kernel 5.3.x available in the repos see:
Code:
$ apt policy pve-kernel-5.3* | grep pve-kernel
pve-kernel-5.3:
pve-kernel-5.3.10-1-pve:
pve-kernel-5.3.1-1-pve:
pve-kernel-5.3.7-1-pve:I do have different problem with kernel 5.3.7-1-pve and my only option is to upgrade to pve-kernel-5.3.10-1-pve and how it will fix the RCOTI.
Hi Kubicz,
You probably know more about this than I, but in my brief experience virtualizing pfsense, the only way to get AES instructions exposed to pfsense is to run it this way. Similarly, I seem to recall enabling "nested" capabilities in virtualbox to provide the same functionality to pfsense running in that hypervisor.
There may be another way, and it's also possible that there's no actual need to "expose" the capability in this manner to get good performance. In fact, many pfsense guides suggest not enabling AES acceleration in pfsense at all as it may introduce an unnecessary context switch or something like that. (I'm probably saying this wrong). The theory being that the AES capabilities of the CPU will be exploited at a "lower" level of operations automatically.
Using the default KVM64 CPU setting setting may actually be fine. I should explore this further.
Yes, that makes sense. I will later check what CPU instructions are presented to FreeBSD underlying pfSense. AES-NI definitelly impacts performance of OpenVPN. Let's dig it deeper.
Sure, it is a must in Nested Virtualization Use Case. Just a little note: in documentation is mentioned that in case of AMD CPU certain line has to be added to VM config file residing at
/etc/pve/qemu-server/{vmid}.conf:
Code:
args: -cpu host,+svmI had an issue in case of Intel CPU, so I digged out there is similar line which resolved RCOTI on one of our servers. It may help someone.
Code:
args: -cpu host,+vmxYes, I've bumped to articles saying not to expose AES-NI instructions to pfSense VM, but still I will try to do some stability and performance comparison of VM in "host" mode running pfSense with AES-NI enabled vs. Intel/AMD CPU mode running pfSense with AES-NI enabled.
Security-wise, it's over my current knowledge, but maybe someone can add some low level detail.
Well, in my case no VM will boot with anything other than 'host'.
Code:
kvm: warning: host doesn't support requested feature: CPUID.80000001H:EDX.nx [bit 20]
kvm: Host doesn't support requested featuresWell, that's an interesting Use Case. Generally KVM is complaining about one CPU flag missing on the host OS level presented to it by the installed CPU. CPU flag is either missing or not supported by QEMU version 6.0, which I doubt if you're not running any "ancient" CPU.
Can you please share output of host's
/proc/cpuinfo?One personal remark: I am trying to avoid UEFI at all levels, because it's unnecessary level of abstraction above the HW causing several practical troubles in certain Use Cases. It's been considered as security risk by design, and because I am not an expert, I am trying to avoid it.
I am not saying it's the RCOTI, but as I do deploy just VMs/LXCs with legacy BIOS, I don't know if UEFI may impact Your Use Case. But BIOS/UEFI settings definitely can. We will see what we can do with Your CPU info.
Here we go:Well, that's an interesting Use Case. Generally KVM is complaining about one CPU flag missing on the host OS level presented to it by the installed CPU. CPU flag is either missing or not supported by QEMU version 6.0, which I doubt if you're not running any "ancient" CPU.
Can you please share output of host's/proc/cpuinfo?
One personal remark: I am trying to avoid UEFI at all levels, because it's unnecessary level of abstraction above the HW causing several practical troubles in certain Use Cases. It's been considered as security risk by design, and because I am not an expert, I am trying to avoid it.
I am not saying it's the RCOTI, but as I do deploy just VMs/LXCs with legacy BIOS, I don't know if UEFI may impact Your Use Case. But BIOS/UEFI settings definitely can. We will see what we can do with Your CPU info.
Code:
#root@dfi: cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 48
model name : AMD RX-427BB with AMD Radeon(tm) R7 Graphics
stepping : 1
microcode : 0x6003106
cpu MHz : 1972.583
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 16
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb bpext ptsc cpb hw_pstate ssbd vmmcall fsgsbase bmi1 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 5390.07
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro [13]
processor : 1
vendor_id : AuthenticAMD
cpu family : 21
model : 48
model name : AMD RX-427BB with AMD Radeon(tm) R7 Graphics
stepping : 1
microcode : 0x6003106
cpu MHz : 2071.665
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 17
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb bpext ptsc cpb hw_pstate ssbd vmmcall fsgsbase bmi1 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 5390.07
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro [13]
processor : 2
vendor_id : AuthenticAMD
cpu family : 21
model : 48
model name : AMD RX-427BB with AMD Radeon(tm) R7 Graphics
stepping : 1
microcode : 0x6003106
cpu MHz : 2035.019
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 2
apicid : 18
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb bpext ptsc cpb hw_pstate ssbd vmmcall fsgsbase bmi1 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 5390.07
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro [13]
processor : 3
vendor_id : AuthenticAMD
cpu family : 21
model : 48
model name : AMD RX-427BB with AMD Radeon(tm) R7 Graphics
stepping : 1
microcode : 0x6003106
cpu MHz : 2041.525
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 2
apicid : 19
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb bpext ptsc cpb hw_pstate ssbd vmmcall fsgsbase bmi1 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold overflow_recov
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 5390.07
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro [13]Here's one VM config that fails to boot with default kvm64:
Code:
bootdisk: scsi0
cores: 2
cpu: kvm64
ide2: none,media=cdrom
memory: 1024
name: orchid
net0: virtio=BE:9F:70:9F:7B:1B,bridge=vmbr0
numa: 0
ostype: l26
scsi0: local:106/vm-106-disk-0.qcow2,size=8G
scsihw: virtio-scsi-pci
smbios1: uuid=14a4cbcf-783a-4f67-b089-65aa0e06de7b
sockets: 1
vmgenid: 12009ccd-22f4-4001-8836-26123768a0d9UEFI has it's own slew of problems, here's one (also mine): https://forum.proxmox.com/threads/unable-to-boot-ovmf-vm-on-proxmox-6.60424/. Sometimes it's unavoidable if you obtain a pre-built image.