FreeNAS kernel panics on initial boot

W

woopla

New Member
Jan 4, 2020
7
2
1
45
Hello,

I've moved a few VMs from XCP-ng over to Proxmox, to be able to use GPU passthrough properly. Alas, my FreeNAS VM has an issue when I start it up: the FreeBSD kernel starts then panics and drops me in the debugger, seems to be coming from an issue with the AT keyboard part. If I just issue a reboot from the debugger, the VM reboots (so it "stays up" as far as KVM is concerned) and FreeNAS boots fine.

Has anybody seen that issue before? I'm using Proxmox 6.1 on a Ryzen platform.

Thanks,

Clément
 
Do you still have this problem? Is there any useful output when the kernel panics?
 
Hi Dominic,

Yes, I still have the problem. FreeNAS sends a daily security e-mail if anything happens, unexpected reboots included - here's the relevant part of the output:
Code: 
> panic: vm_wait in early boot
> cpuid = 1
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff8288f170
> vpanic() at vpanic+0x177/frame 0xffffffff8288f1d0
> panic() at panic+0x43/frame 0xffffffff8288f230
> vm_page_alloc_contig() at vm_page_alloc_contig/frame 0xffffffff8288f260
> vm_page_alloc_after() at vm_page_alloc_after+0xc0/frame 0xffffffff8288f2b0
> vm_page_grab_pages() at vm_page_grab_pages+0x180/frame 0xffffffff8288f330
> vm_thread_new() at vm_thread_new+0x1a8/frame 0xffffffff8288f470
> thread_alloc() at thread_alloc+0x31/frame 0xffffffff8288f490
> kthread_add() at kthread_add+0x43/frame 0xffffffff8288f530
> kproc_kthread_add() at kproc_kthread_add+0xb3/frame 0xffffffff8288f650
> intr_event_add_handler() at intr_event_add_handler+0x1ac/frame 0xffffffff8288f6c0
> intr_add_handler() at intr_add_handler+0x75/frame 0xffffffff8288f720
> nexus_setup_intr() at nexus_setup_intr+0x8a/frame 0xffffffff8288f770
> bus_setup_intr() at bus_setup_intr+0xa2/frame 0xffffffff8288f7d0
> atkbdattach() at atkbdattach+0xc1/frame 0xffffffff8288f820
> device_attach() at device_attach+0x3b8/frame 0xffffffff8288f870
> bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8288f8a0
> atkbdc_isa_attach() at atkbdc_isa_attach+0x17e/frame 0xffffffff8288f8e0
> device_attach() at device_attach+0x3b8/frame 0xffffffff8288f930
> bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8288f960
> acpi_attach() at acpi_attach+0xe39/frame 0xffffffff8288fa10
> device_attach() at device_attach+0x3b8/frame 0xffffffff8288fa60
> bus_generic_attach() at bus_generic_attach+0x5c/frame 0xffffffff8288fa90
> device_attach() at device_attach+0x3b8/frame 0xffffffff8288fae0
> bus_generic_new_pass() at bus_generic_new_pass+0x118/frame 0xffffffff8288fb10
> root_bus_configure() at root_bus_configure+0x77/frame 0xffffffff8288fb40
> configure() at configure+0x9/frame 0xffffffff8288fb50
> mi_startup() at mi_startup+0x118/frame 0xffffffff8288fb70
> btext() at btext+0x2c
> KDB: enter: panic

The remainder of the e-mail is the regular boot after I issue the reboot command inside the debugger.

Here's my config file for this VM:
Code: 
agent: 0
balloon: 256
bootdisk: ide0
cores: 2
cpu: host
ide0: local-ssd:100/vm-100-disk-0.qcow2,size=20G
ide2: none,media=cdrom
memory: 8192
name: FreeNAS
net0: e1000=36:83:91:B3:41:C8,bridge=vmbr0
numa: 0
onboot: 1
ostype: other
sata0: /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M4DX0YH3,size=1953514584K
sata1: /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M7HFR6L2,size=1953514584K
sata2: /dev/disk/by-id/ata-WDC_WD20EFRX-68AX9N0_WD-WMC301429022,size=1953514584K
sata3: /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WMC4M1766270,size=1953514584K
sata4: /dev/disk/by-id/ata-WDC_WD20EFRX-68EUZN0_WD-WMC4M1788276,size=1953514584K
scsihw: virtio-scsi-pci
smbios1: uuid=affb66ac-20ea-4c39-b7bb-6d83047f88e1
sockets: 1
startup: order=2,up=300
tablet: 0
vga: std
vmgenid: b817437e-1b73-42bb-a00f-8b1993a1ec0d

Thanks,

Clément
 
Have you tried changing the cpu type of the virtual machine?
 
I can't - selecting anything else cause qemu to fail starting VMs - I would have to create a test one to give you the exact error message, but it boils down to some feature flags not being present on the CPU (it's a Ryzen 1700X).
 
Here's the error message when I set the CPU type to kvm64 (the default), qemu64 or EPYC:
Code: 
kvm: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
kvm: Host doesn't support requested features
TASK ERROR: start failed: QEMU exited with code 1
 
Could you post the output of 'lscpu' and 'dmesg | grep -e apic -e avic -e irq -e interrupt -e iommu' on your PVE host?
 
  • Like
Reactions: Dominic
Here you go for lscpu:
Code: 
root@medusa:~# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       43 bits physical, 48 bits virtual
CPU(s):              16
On-line CPU(s) list: 0-15
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               1
Model name:          AMD Ryzen 7 1700X Eight-Core Processor
Stepping:            1
CPU MHz:             2898.303
CPU max MHz:         3400.0000
CPU min MHz:         2200.0000
BogoMIPS:            6786.11
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           64K
L2 cache:            512K
L3 cache:            8192K
NUMA node0 CPU(s):   0-15
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
 
And now for dmesg:
Code: 
root@medusa:~# dmesg | grep -i -e apic -e avic -e irq -e interrupt -e iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.13-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1
[    0.000000] ACPI: APIC 0x00000000C13EBB28 00015E (v03 ALASKA A M I    01072009 AMI  00010013)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] system APIC only can use physical flat
[    0.000000] Setting APIC routing to physical flat.
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] system APIC only can use physical flat
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 17, version 33, address 0xfec00000, GSI 0-23
[    0.000000] IOAPIC[1]: apic_id 18, version 33, address 0xfec01000, GSI 24-55
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.3.13-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1
[    0.000000] NR_IRQS: 524544, nr_irqs: 1224, preallocated irqs: 16
[    0.000000] APIC: Switch to symmetric I/O mode setup
[    0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.181737] ACPI: Using IOAPIC for interrupt routing
[    0.200239] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 5 7 10 11 14 15) *0
[    0.200299] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 5 7 10 11 14 15) *0
[    0.200353] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 5 7 10 11 14 15) *0
[    0.200418] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 5 7 10 11 14 15) *0
[    0.200477] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 5 7 10 11 14 15) *0
[    0.200526] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 7 10 11 14 15) *0
[    0.200574] ACPI: PCI Interrupt Link [LNKG] (IRQs 4 5 7 10 11 14 15) *0
[    0.200623] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 7 10 11 14 15) *0
[    0.201178] PCI: Using ACPI for IRQ routing
[    0.205965] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.699884] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.700377] pci 0000:00:01.0: Adding to iommu group 0
[    0.700403] pci 0000:00:01.0: Using iommu direct mapping
[    0.700494] pci 0000:00:01.1: Adding to iommu group 1
[    0.700514] pci 0000:00:01.1: Using iommu direct mapping
[    0.700603] pci 0000:00:01.3: Adding to iommu group 2
[    0.700623] pci 0000:00:01.3: Using iommu direct mapping
[    0.700708] pci 0000:00:02.0: Adding to iommu group 3
[    0.700728] pci 0000:00:02.0: Using iommu direct mapping
[    0.700821] pci 0000:00:03.0: Adding to iommu group 4
[    0.700841] pci 0000:00:03.0: Using iommu direct mapping
[    0.700927] pci 0000:00:03.1: Adding to iommu group 5
[    0.700946] pci 0000:00:03.1: Using iommu direct mapping
[    0.701031] pci 0000:00:03.2: Adding to iommu group 6
[    0.701051] pci 0000:00:03.2: Using iommu direct mapping
[    0.701137] pci 0000:00:04.0: Adding to iommu group 7
[    0.701156] pci 0000:00:04.0: Using iommu direct mapping
[    0.701244] pci 0000:00:07.0: Adding to iommu group 8
[    0.701263] pci 0000:00:07.0: Using iommu direct mapping
[    0.701348] pci 0000:00:07.1: Adding to iommu group 9
[    0.701367] pci 0000:00:07.1: Using iommu direct mapping
[    0.701455] pci 0000:00:08.0: Adding to iommu group 10
[    0.701474] pci 0000:00:08.0: Using iommu direct mapping
[    0.701560] pci 0000:00:08.1: Adding to iommu group 11
[    0.701580] pci 0000:00:08.1: Using iommu direct mapping
[    0.701668] pci 0000:00:14.0: Adding to iommu group 12
[    0.701687] pci 0000:00:14.0: Using iommu direct mapping
[    0.701698] pci 0000:00:14.3: Adding to iommu group 12
[    0.701809] pci 0000:00:18.0: Adding to iommu group 13
[    0.701828] pci 0000:00:18.0: Using iommu direct mapping
[    0.701840] pci 0000:00:18.1: Adding to iommu group 13
[    0.701853] pci 0000:00:18.2: Adding to iommu group 13
[    0.701864] pci 0000:00:18.3: Adding to iommu group 13
[    0.701875] pci 0000:00:18.4: Adding to iommu group 13
[    0.701886] pci 0000:00:18.5: Adding to iommu group 13
[    0.701897] pci 0000:00:18.6: Adding to iommu group 13
[    0.701908] pci 0000:00:18.7: Adding to iommu group 13
[    0.701996] pci 0000:01:00.0: Adding to iommu group 14
[    0.702015] pci 0000:01:00.0: Using iommu direct mapping
[    0.702124] pci 0000:02:00.0: Adding to iommu group 15
[    0.702144] pci 0000:02:00.0: Using iommu direct mapping
[    0.702168] pci 0000:02:00.1: Adding to iommu group 15
[    0.702192] pci 0000:02:00.2: Adding to iommu group 15
[    0.702201] pci 0000:03:00.0: Adding to iommu group 15
[    0.702211] pci 0000:03:02.0: Adding to iommu group 15
[    0.702221] pci 0000:03:03.0: Adding to iommu group 15
[    0.702232] pci 0000:03:04.0: Adding to iommu group 15
[    0.702243] pci 0000:03:06.0: Adding to iommu group 15
[    0.702253] pci 0000:03:07.0: Adding to iommu group 15
[    0.702272] pci 0000:07:00.0: Adding to iommu group 15
[    0.702290] pci 0000:08:00.0: Adding to iommu group 15
[    0.702388] pci 0000:0a:00.0: Adding to iommu group 16
[    0.702407] pci 0000:0a:00.0: Using iommu direct mapping
[    0.702430] pci 0000:0a:00.1: Adding to iommu group 16
[    0.702533] pci 0000:0b:00.0: Adding to iommu group 17
[    0.702552] pci 0000:0b:00.0: Using iommu direct mapping
[    0.702577] pci 0000:0b:00.1: Adding to iommu group 17
[    0.702666] pci 0000:0c:00.0: Adding to iommu group 18
[    0.702685] pci 0000:0c:00.0: Using iommu direct mapping
[    0.702773] pci 0000:0c:00.2: Adding to iommu group 19
[    0.702793] pci 0000:0c:00.2: Using iommu direct mapping
[    0.702883] pci 0000:0c:00.3: Adding to iommu group 20
[    0.702902] pci 0000:0c:00.3: Using iommu direct mapping
[    0.702991] pci 0000:0d:00.0: Adding to iommu group 21
[    0.703013] pci 0000:0d:00.0: Using iommu direct mapping
[    0.703102] pci 0000:0d:00.2: Adding to iommu group 22
[    0.703122] pci 0000:0d:00.2: Using iommu direct mapping
[    0.703210] pci 0000:0d:00.3: Adding to iommu group 23
[    0.703230] pci 0000:0d:00.3: Using iommu direct mapping
[    0.703413] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.703415]  PPR NX GT IA GA PC GA_vAPIC
[    0.703416] AMD-Vi: Interrupt remapping enabled
[    0.703417] AMD-Vi: Virtual APIC enabled
[    0.703864] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.712807] pcieport 0000:00:01.1: PME: Signaling with IRQ 28
[    0.712873] pcieport 0000:00:01.1: AER: enabled with IRQ 28
[    0.713030] pcieport 0000:00:01.3: PME: Signaling with IRQ 29
[    0.713081] pcieport 0000:00:01.3: AER: enabled with IRQ 29
[    0.713257] pcieport 0000:00:03.1: PME: Signaling with IRQ 30
[    0.713295] pcieport 0000:00:03.1: AER: enabled with IRQ 30
[    0.713473] pcieport 0000:00:03.2: PME: Signaling with IRQ 31
[    0.713517] pcieport 0000:00:03.2: AER: enabled with IRQ 31
[    0.713692] pcieport 0000:00:07.1: PME: Signaling with IRQ 32
[    0.713732] pcieport 0000:00:07.1: AER: enabled with IRQ 32
[    0.713962] pcieport 0000:00:08.1: PME: Signaling with IRQ 33
[    0.713999] pcieport 0000:00:08.1: AER: enabled with IRQ 33
[    0.717652] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.738536] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    0.871537] rtc_cmos 00:01: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    1.076024] ata1: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780100 irq 56
[    1.076026] ata2: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780180 irq 56
[    1.076028] ata3: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780200 irq 56
[    1.076030] ata4: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780280 irq 56
[    1.076031] ata5: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780300 irq 56
[    1.076032] ata6: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780380 irq 56
[    1.076034] ata7: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780400 irq 56
[    1.076035] ata8: SATA max UDMA/133 abar m131072@0xf6780000 port 0xf6780480 irq 56
[    1.076471] ata9: SATA max UDMA/133 abar m4096@0xf6808000 port 0xf6808100 irq 63
[    1.100646] igb 0000:08:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
[    6.715272] SVM: AVIC enabled
[59722.197387] perf: interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
 
What happens if you write into your VM config (/etc/pve/qemu-server/VMID.conf) the following line
Code: 
args: -cpu kvm64,-x2apic
(and remove the line "cpu: host")?
 
Thanks Dominic, that appears to be working :D

However I'll keep monitoring, because it did reboot OK one time with the previous settings - hopefully that'll prove more stable this time around.
 
  • Like
Reactions: Stefan_R and Dominic
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom