Hi,
I have a server running Proxmox 5.3.11. The kworker always uses too much CPU that caused the server hangs. After debugging, I see a lot of "kvm Tained" in the dmesg log. Some example:
My PVE version:
Anyone here facing the same issue like me and do you have any solution?
Thanks.
I have a server running Proxmox 5.3.11. The kworker always uses too much CPU that caused the server hangs. After debugging, I see a lot of "kvm Tained" in the dmesg log. Some example:
Code:
....
[661233.727943] NMI backtrace for cpu 4
[661233.727945] CPU: 4 PID: 807 Comm: rs:main Q:Reg Tainted: G D 4.15.18-11-pve #1
[661233.727945] Hardware name: MSI MS-7816/H87-G43 (MS-7816), BIOS V2.14B14 07/13/2018
[661233.727946] RIP: 0033:0x7f4b9aae98ef
[661233.727947] RSP: 002b:00007f4b9941dab0 EFLAGS: 00000202
[661233.727948] RAX: 0000000000000085 RBX: 00007f4b94002070 RCX: 0000000000003987
[661233.727949] RDX: fffffffffffffd90 RSI: 0000000000000000 RDI: 00007f4b94000020
[661233.727949] RBP: 00007f4b94000020 R08: 000055b74ab7f510 R09: 0000000000000000
[661233.727950] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[661233.727951] R13: 0000000000000270 R14: 00007f4b940022e0 R15: 0000000000000080
[661233.727952] FS: 00007f4b9941e700(0000) GS:ffff9c29deb00000(0000) knlGS:0000000000000000
[661233.727953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[661233.727953] CR2: ffffb001ac5b877b CR3: 00000007f5d08001 CR4: 00000000001626e0
[661233.727954] NMI backtrace for cpu 0
[661233.727956] CPU: 0 PID: 24709 Comm: kvm Tainted: G D 4.15.18-11-pve #1
[661233.727956] Hardware name: MSI MS-7816/H87-G43 (MS-7816), BIOS V2.14B14 07/13/2018
[661233.727978] RIP: 0010:paging64_walk_addr_generic+0x1ca/0x7d0 [kvm]
[661233.727978] RSP: 0018:ffffaf4b43ddb960 EFLAGS: 00000246
[661233.727979] RAX: 0000000000000003 RBX: 0000000000000004 RCX: 0000000000804063
[661233.727980] RDX: 0000000000000003 RSI: 8000000000804063 RDI: 0000000000000004
[661233.727981] RBP: ffffaf4b43ddb9f8 R08: 0000000000000003 R09: 0000000000000001
[661233.727981] R10: 0000000000000000 R11: ffff9c2925c20000 R12: 00000000000001aa
[661233.727982] R13: 0000000000804063 R14: ffff9c2947400380 R15: ffffaf4b43ddba08
[661233.727983] FS: 00007f6d173ff700(0000) GS:ffff9c29dea00000(0000) knlGS:000000007f3ec000
[661233.727983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[661233.727984] CR2: ffffb001ac5b877b CR3: 00000007f2c12006 CR4: 00000000001626f0
[661233.727984] Call Trace:
[661233.727998] ? kvm_irq_delivery_to_apic+0x1e8/0x2b0 [kvm]
[661233.728010] paging64_gva_to_gpa+0x44/0xb0 [kvm]
[661233.728020] ? kvm_vcpu_mmap+0x20/0x20 [kvm]
[661233.728024] ? vmx_vcpu_load+0x6a/0x300 [kvm_intel]
[661233.728027] ? vmx_vcpu_load+0x248/0x300 [kvm_intel]
[661233.728035] ? kvm_io_bus_get_first_dev+0x58/0x110 [kvm]
[661233.728045] ? kvm_io_bus_read+0x48/0x110 [kvm]
[661233.728047] ? __update_load_avg_se.isra.38+0x1bc/0x1d0
[661233.728048] ? __update_load_avg_se.isra.38+0x1bc/0x1d0
[661233.728059] kvm_fetch_guest_virt+0x6f/0xe0 [kvm]
[661233.728070] __do_insn_fetch_bytes+0x1ae/0x220 [kvm]
[661233.728078] x86_decode_insn+0x79d/0x1310 [kvm]
[661233.728080] ? vmx_get_cs_db_l_bits+0x1c/0x40 [kvm_intel]
[661233.728087] x86_emulate_instruction+0x13c/0x6e0 [kvm]
[661233.728089] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel]
[661233.728090] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel]
[661233.728092] handle_apic_access+0x4e/0x70 [kvm_intel]
[661233.728094] vmx_handle_exit+0xb5/0x1560 [kvm_intel]
[661233.728096] ? vmexit_fill_RSB+0x10/0x40 [kvm_intel]
[661233.728098] ? vmx_vcpu_run+0x418/0x5e0 [kvm_intel]
[661233.728105] kvm_arch_vcpu_ioctl_run+0x95c/0x16d0 [kvm]
[661233.728107] ? futex_wake+0x90/0x170
[661233.728114] ? kvm_arch_vcpu_load+0x68/0x250 [kvm]
[661233.728120] kvm_vcpu_ioctl+0x339/0x620 [kvm]
[661233.728128] ? kvm_vcpu_ioctl+0x339/0x620 [kvm]
[661233.728130] do_vfs_ioctl+0xa6/0x620
[661233.728141] ? kvm_on_user_return+0x70/0xa0 [kvm]
[661233.728142] SyS_ioctl+0x79/0x90
[661233.728144] ? exit_to_usermode_loop+0xa5/0xd0
[661233.728146] do_syscall_64+0x73/0x130
[661233.728147] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[661233.728148] RIP: 0033:0x7f6e6140c017
[661233.728149] RSP: 002b:00007f6d173fc538 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[661233.728150] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f6e6140c017
[661233.728150] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000032
[661233.728151] RBP: 0000000000000000 R08: 00007f6e53861ef0 R09: 000001a64a020188
[661233.728152] R10: 000000003b9aca00 R11: 0000000000000246 R12: 00007f6e53be1000
[661233.728152] R13: 00007f6e79d28000 R14: 0000000000000000 R15: 00007f6e53be1000
[661233.728153] Code: 00 00 00 41 8b 3f 4c 89 e8 44 89 e9 48 c1 e8 07 83 e0 01 8d 57 ff 4c 63 c2 4d 8d 14 80 4c 01 d0 4d 89 ea 4d 23 94 c6 e0 00 00 00 <49> 8b 86 30 01 00 00 48 d3 e8 83 e0 01 49 09 c2 0f 85 d6 01 00
[661233.728174] NMI backtrace for cpu 2 skipped: idling at intel_idle+0x7b/0x130
[661233.728175] NMI backtrace for cpu 6
[661233.728177] CPU: 6 PID: 3873 Comm: kvm Tainted: G D 4.15.18-11-pve #1
[661233.728177] Hardware name: MSI MS-7816/H87-G43 (MS-7816), BIOS V2.14B14 07/13/2018
[661233.728180] RIP: 0010:intel_guest_get_msrs+0x18/0x70
[661233.728181] RSP: 0018:ffffaf4b4811fc78 EFLAGS: 00000086
[661233.728182] RAX: ffff9c29deb8f3a0 RBX: ffff9c29b27dd628 RCX: ffff9c29b27d8000
[661233.728183] RDX: 0000000000000004 RSI: 000063ddc0000000 RDI: ffffaf4b4811fc9c
[661233.728183] RBP: ffffaf4b4811fc78 R08: 0000000000000000 R09: 0000000000000000
[661233.728184] R10: ffffaf4b4811fdd8 R11: 0000000000000000 R12: 0000000000000000
[661233.728185] R13: ffff9c29b27d8000 R14: ffff9c29b2f952b0 R15: 0000000000000001
[661233.728186] FS: 00007f8195fff700(0000) GS:ffff9c29deb80000(0000) knlGS:00000000fe520000
[661233.728186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[661233.728187] CR2: 0000000009da3764 CR3: 00000007f2ec4004 CR4: 00000000001626e0
[661233.728188] Call Trace:
[661233.728189] perf_guest_get_msrs+0x1a/0x30
[661233.728194] atomic_switch_perf_msrs+0x2d/0xa0 [kvm_intel]
[661233.728197] vmx_vcpu_run+0x13a/0x5e0 [kvm_intel]
[661233.728210] kvm_arch_vcpu_ioctl_run+0x841/0x16d0 [kvm]
[661233.728212] ? futex_wake+0x90/0x170
[661233.728223] ? kvm_arch_vcpu_load+0x68/0x250 [kvm]
[661233.728229] kvm_vcpu_ioctl+0x339/0x620 [kvm]
[661233.728236] ? kvm_vcpu_ioctl+0x339/0x620 [kvm]
[661233.728237] do_vfs_ioctl+0xa6/0x620
[661233.728244] ? kvm_on_user_return+0x70/0xa0 [kvm]
[661233.728245] SyS_ioctl+0x79/0x90
[661233.728247] do_syscall_64+0x73/0x130
[661233.728248] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[661233.728249] RIP: 0033:0x7f82e0264017
[661233.728249] RSP: 002b:00007f8195ffc538 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[661233.728250] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f82e0264017
[661233.728251] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000033
[661233.728251] RBP: 0000000000000000 R08: 00007f82d2461ef0 R09: 00000000000000ff
[661233.728251] R10: 00007f82f8b81000 R11: 0000000000000246 R12: 00007f82d27e1000
[661233.728252] R13: 00007f82f8b80000 R14: 0000000000000000 R15: 00007f82d27e1000
[661233.728253] Code: 07 00 00 00 00 31 c0 c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 c7 c0 a0 f3 00 00 48 89 e5 65 48 03 05 30 47 20 64 <48> 8b 90 80 0c 00 00 48 8b 35 6a fc 64 01 48 05 88 0c 00 00 48
...My PVE version:
Code:
root@hostname~ # pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-2
pve-kernel-4.15.18-11-pve: 4.15.18-33
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-47
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-38
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-34
pve-docs: 5.3-3
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-2
pve-xtermjs: 3.10.1-1
qemu-server: 5.0-46
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3Anyone here facing the same issue like me and do you have any solution?
Thanks.
