Hi everyone,
I'm encountering an issue with my OPNsense VM after Proxmox backup operations. Here's my setup and the specific problem:
Setup:
- Running multiple VMs in Proxmox with nightly backups using Proxmox Backup Server
- Backups are configured in shutdown mode (VMs shut down before backup)
- Two VMs use PCIe passthrough:
1. One VM with GPU passthrough (working fine)
2. OPNsense VM with WAN NIC passthrough (having issues)
Issue:
The OPNsense VM fails to properly start after the backup process completes. However, if I manually shut down and start the VM through the Proxmox GUI, it boots normally. This issue only occurs during the automated backup/restart process.
The VM appears to get stuck during the boot process, with the console showing FreeBSD boot messages but not completing the startup sequence.
The VM boots up fine if i manually shut it down through the Proxmox GUI and start it up but this process seems to not work as well when done automatically during backups.
Questions:
1. Has anyone encountered similar issues with PCIe passthrough devices after backup operations?
2. Are there any known workarounds or configuration changes that might help resolve this?
Logs from OPNsense VM console:
VM Configuration:
I'm encountering an issue with my OPNsense VM after Proxmox backup operations. Here's my setup and the specific problem:
Setup:
- Running multiple VMs in Proxmox with nightly backups using Proxmox Backup Server
- Backups are configured in shutdown mode (VMs shut down before backup)
- Two VMs use PCIe passthrough:
1. One VM with GPU passthrough (working fine)
2. OPNsense VM with WAN NIC passthrough (having issues)
Issue:
The OPNsense VM fails to properly start after the backup process completes. However, if I manually shut down and start the VM through the Proxmox GUI, it boots normally. This issue only occurs during the automated backup/restart process.
The VM appears to get stuck during the boot process, with the console showing FreeBSD boot messages but not completing the startup sequence.
The VM boots up fine if i manually shut it down through the Proxmox GUI and start it up but this process seems to not work as well when done automatically during backups.
Questions:
1. Has anyone encountered similar issues with PCIe passthrough devices after backup operations?
2. Are there any known workarounds or configuration changes that might help resolve this?
Logs from OPNsense VM console:
Code:
Loading kernel...
/boot/kernel/kernel text=0x180c00 text=0xdfd718 text=0x455014 data=0x180+0xe80 data=0x196980+0x469680 0x8+0x19f5f0+0x8+0x1c4187|
Loading configured modules...
/boot/kernel/if_enc.ko size 0x4be0 at 0x2164000
/boot/entropy size=0x1000
/boot/kernel/pflog.ko size 0x3c10 at 0x216a000
loading required module 'pf'
/boot/kernel/pf.ko size 0x8c548 at 0x216e000
/boot/kernel/carp.ko size 0xfba8 at 0x21fb000
/boot/kernel/if_bridge.ko size 0xea58 at 0x220b000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x8930 at 0x221a000
/boot/kernel/if_gre.ko size 0xaa30 at 0x2223000
/boot/kernel/pfsync.ko size 0x11a18 at 0x222e000
/etc/hostid size=0x25
/boot/kernel/if_lagg.ko size 0x165f0 at 0x2240000
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko size 0x3540 at 0x2257000
staging 0x73000000 (not copying) tramp 0x7c9b2000 PT4 0x7c9a9000
Start @ 0xffffffff80381000 ...
EFI framebuffer information:
addr, size 0x80000000, 0x3e8000
dimensions 1280 x 800
stride 1280
masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p6 stable/24.7-n267981-8375762712f SMP amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(efifb): resolution 1280x800
CPU: Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz (2300.13-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2
Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0xfffab223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x21<LAHF,ABM>
Structured Extended Features=0x7ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID>
Structured Extended Features2=0x4<UMIP>
Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
XSAVE Features=0x1<XSAVEOPT>
IA32_ARCH_CAPS=0x4000048<SKIP_L1DFL_VME>
AMD Extended Feature Extensions ID EBX=0x100d000<IBPB,IBRS,STIBP,SSBD>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
real memory = 12582912000 (12000 MB)
avail memory = 12089155584 (11529 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BOCHS BXPC >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 1.1> irqs 0-23
Launching APs: 3 5 7 2 1 6 4
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
kvmclock0: <KVM paravirtual clock>
Timecounter "kvmclock" frequency 1000000000 Hz quality 975
kvmclock0: registered as a time-of-day clock, resolution 0.000001s
smbios0: <System Management BIOS> at iomem 0x7e9d4000-0x7e9d401e
smbios0: Version: 2.8, BCD Revision: 2.8
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <BOCHS BXPC>
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> mem 0x80000000-0x80ffffff,0x8224b000-0x8224bfff at device 1.0 on pci0
vgapci0: Boot video device
uhci0: <Intel 82801I (ICH9) USB controller> port 0x9100-0x911f irq 16 at device 26.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: <Intel 82801I (ICH9) USB controller> port 0x90e0-0x90ff irq 17 at device 26.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2: <Intel 82801I (ICH9) USB controller> port 0x90c0-0x90df irq 18 at device 26.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0x8224a000-0x8224afff irq 19 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
hdac0: <Intel 82801I HDA Controller> mem 0x82240000-0x82243fff irq 16 at device 27.0 on pci0
pcib1: <ACPI PCI-PCI bridge> mem 0x82249000-0x82249fff irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) I210 (Copper)> port 0x8000-0x801f mem 0x82000000-0x8207ffff,0x82080000-0x82083fff irq 16 at device 0.0 on pci1VM Configuration:
Code:
root@proxmox:~# cat /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0
cores: 8
cpu: host,flags=+aes
efidisk0: vms:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:09:00,pcie=1
machine: q35
memory: 12000
meta: creation-qemu=9.0.2,ctime=1734042654
name: opnsense
net0: virtio=BC:24:11:05:22:D2,bridge=vmbr0,queues=8
numa: 0
onboot: 1
ostype: other
protection: 1
scsi0: vms:vm-100-disk-1,discard=on,iothread=1,size=64G,ssd=1
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=d66bcb58-5516-4010-9b46-7aba47775c09
sockets: 1
startup: order=1
tags: management
vmgenid: a900f607-09cc-4783-b4a0-dde7c4c7c219