VM Dumping Stack Trace

A

AndyRed

Member
Aug 16, 2019
18
5
23
56
Greetings All:

I have a VM that is dumping a stack trace and rebooting. Please see the following output from this VM:

Jan 30 23:52:54 192.168.2.43 BUG: unable to handle kernel paging request at ffffffff8103f182
Jan 30 23:52:54 192.168.2.43 IP: kvm_kick_cpu+0x22/0x30
Jan 30 23:52:54 192.168.2.43 Oops: 0003 [#1] SMP PTI
Jan 30 23:52:54 192.168.2.43 CPU: 0 PID: 10607 Comm: rrd-vs_total-up Tainted: P O 4.14.137 #1
Jan 30 23:52:54 192.168.2.43 task: ffff888070ba0b40 task.stack: ffff88806f334000
Jan 30 23:52:54 192.168.2.43 RIP: 0010:kvm_kick_cpu+0x22/0x30
Jan 30 23:52:54 192.168.2.43 RSP: 0018:ffff88806f337c68 EFLAGS: 00010046
Jan 30 23:52:54 192.168.2.43 RAX: 0000000000000005 RBX: 0000000000000000 RCX: 0000000000000002
Jan 30 23:52:54 192.168.2.43 RDX: ffff88807c900000 RSI: ffff88807ca79740 RDI: 0000000000000002
Jan 30 23:52:54 192.168.2.43 RBP: ffff88806f337c70 R08: 00000000000000ec R09: 0000000000000100
Jan 30 23:52:54 192.168.2.43 R10: 0000000000000100 R11: 0000000000000001 R12: 000000000000001f
Jan 30 23:52:54 192.168.2.43 R13: ffff88807ca81000 R14: ffffea0000075278 R15: ffff88806f1cb000

... more to come in a follow up comment.
 
--- snipping ---

Jan 30 23:52:54 192.168.2.43 FS: 0000000000000000(0000) GS:ffff88807c800000(0000) knlGS:0000000000000000
Jan 30 23:52:54 192.168.2.43 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 30 23:52:54 192.168.2.43 CR2: ffffffff8103f182 CR3: 000000000220a000 CR4: 00000000000006b0
Jan 30 23:52:54 192.168.2.43 __pv_queued_spin_unlock_slowpath+0x9b/0xd0
Jan 30 23:52:54 192.168.2.43 __raw_callee_save___pv_queued_spin_unlock_slowpath+0x15/0x24
Jan 30 23:52:54 192.168.2.43 .slowpath+0x9/0x15
Jan 30 23:52:54 192.168.2.43 _raw_spin_unlock_irqrestore+0x9/0x20
Jan 30 23:52:54 192.168.2.43 release_pages+0x20a/0x300
Jan 30 23:52:54 192.168.2.43 tlb_flush_mmu_free+0x3a/0x60
Jan 30 23:52:54 192.168.2.43 arch_tlb_finish_mmu+0x42/0x70
Jan 30 23:52:54 192.168.2.43 tlb_finish_mmu+0x1e/0x30
Jan 30 23:52:54 192.168.2.43 exit_mmap+0xd5/0x1a0
Jan 30 23:52:54 192.168.2.43 mmput+0x16/0xb0
Jan 30 23:52:54 192.168.2.43 do_exit+0x209/0xa60
Jan 30 23:52:54 192.168.2.43 ? __do_page_fault+0x1b5/0x450

--- snipping ---

What's interesting is that Proxmox shows the uptime of the VM (commercial load balancer) to be a certain amount of days, but yet within the actual VM, uptime reflects the point at which the VM rebooted. So, Prox doesn't show the reboot of the VM, but the VM clearly boots. Exploring the stack dump of the VM, there is no pattern in the processes on the CPU at the time (rrd-tool and l7flow for example) and kernel unable to do something related to paging. This following link suggests it is a known issue in kvm/qemu - Re: KVM kernel BUG: unable to handle kernel paging request in kvm_kick_cpu — Linux KVM (spinics.net). There are a few other links I've found relating to the paging request.

Thoughts?

Andy
 
For added context, I have two OVS-based VM's converted to qcow2 format and a native KVM VM imported and all three exhibit the same behavior.
 
Hi,
so the stack trace happens within the VMs? What kernel/OS is running in the VMs? What's the output of pveversion -v and qm config <ID> with an ID of an affected VM? What CPU model does the host have?
 
Hi Fiona:

Yes, the stack trace happens within the VM itself. It's a hardened linux appliance so the kernel is custom as is the OS to some degree.

CPU(s)

32 x AMD EPYC 7313P 16-Core Processor (1 Socket)
Kernel Version

Linux 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z)
PVE Manager Version

pve-manager/7.3-4/d69b70d4

root@pve:[~]: pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-helper: 7.3-2
pve-kernel-5.15: 7.3-1
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u2
proxmox-backup-client: 2.3.2-1
proxmox-backup-file-restore: 2.3.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve3

root@pve:[~]: qm config 110
boot: order=scsi0
cores: 4
memory: 2048
name: LM
net0: virtio=6A:89:85:C6:02:1A,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsi0: local-lvm:vm-110-disk-0,size=16388M
scsihw: virtio-scsi-pci #to test on one of the other VM's, I changed the SCSI Controller to VMware PVSCSI, but same result in the dump
smbios1: uuid=0aa36a54-d563-4ea3-873e-4f539dfa60a7
sockets: 1
vmgenid: 8e335662-17cc-41d5-85a5-125c87c7a1d6
 
Last edited:
AndyRed said:
Hi Fiona:

Yes, the stack trace happens within the VM itself. It's a hardened linux appliance so the kernel is custom as is the OS to some degree.
Click to expand...
The mail you linked mentions that
Code: 
> > kernel configuration needs to include PARAVIRT_SPINLOCKS = y; if set
> > to n, the kernel bug is not reproducible.

> > -smp option of qemu needs to be larger than 1 in the script above; it
> > is some kind of concurrency bug.
Does your guest kernel use that configuration option? If you can modify/recompile the guest kernel, you might want to try that. The second alternative seems to be assigning only one CPU to the guest :/

Otherwise, I'd suggest trying kernel 6.1 on the host and see if the issue still occurs.

Since the mail you mentioned is about an older kernel and older QEMU, I don't think there's much point in trying these, but if you want to make sure it's not some kind of other regression, you could do that too.
 
Thanks much, Fiona!

Bit of an update …

CPU of host server: 32 x AMD EPYC 7313P 16-Core Processor (1 Socket)
Kernel: Linux 6.2.6-1-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.6-1 (2023-03-14T17:08Z)
PVE MGR V: pve-manager/7.4-3/9002ab8a

Looking at the VM boot log I see the following …

Linux version 4.14.137 (jenkins@fb6847e20b50) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)) #1 SMP Wed Mar 22 05:13:04 UTC 2023
Command line: root=/dev/ram0 ide=nodma ro acpi=noirq initrd=initrd.gz bonding.max_bonds=0 B100DISK=/dev/sda1 BOOT_IMAGE=linux
KERNEL supported cpus:
Intel GenuineIntel
CPU: vendor_id 'AuthenticAMD' unknown, using generic init.
CPU: Your system may be unstable.
x86/fpu: x87 FPU will use FXSAVE
e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x000000007ffd9fff] usable
BIOS-e820: [mem 0x000000007ffda000-0x000000007fffffff] reserved
BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved
NX (Execute Disable) protection: active
SMBIOS 2.8 present.
DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
Hypervisor detected: KVM

I know that the VM appliance supports Intel ‘only’ and I’m using AMD, so I’m curious if even in generic init, this is the issue. Any suggestions to work around this for testing?
 
You could try changing the CPU type of the VM (VM > Hardware > Processors > Edit > Type in the UI) to an Intel processor, but it needs to be one that is compatible with the actual processor. IIRC, @spirit suggested Nehalem for live migrations between AMD and Intel, so that might be a good one to try. You could also try using CPU type host instead. While it will still be detected as AMD of course, it might be worth a try.
 
Many thanks for the reply/suggestion. I've moved to Nehalem and will report back ...

Linux version 4.14.137 (jenkins@fb6847e20b50) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)) #1 SMP Wed Mar 22 05:13:04 UTC 2023
Command line: root=/dev/ram0 ide=nodma ro acpi=noirq initrd=initrd.gz bonding.max_bonds=0 B100DISK=/dev/sda1 BOOT_IMAGE=linux
KERNEL supported cpus:
Intel GenuineIntel
x86/fpu: x87 FPU will use FXSAVE
e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x000000007ffd9fff] usable
BIOS-e820: [mem 0x000000007ffda000-0x000000007fffffff] reserved
BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
NX (Execute Disable) protection: active
SMBIOS 2.8 present.
DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
Hypervisor detected: KVM
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back