PVE kernel stuck

[Alibek]

Today one of the servers is stuck, some time after the next message:

Feb 17 04:15:11 hostA kernel: [472111.800882] WARNING: CPU: 41 PID: 0 at net/sched/sch_generic.c:323 dev_watchdog+0x222/0x230
Feb 17 04:15:11 hostA kernel: [472111.801110] R13: ffff974a19760000 R14: ffff974a19760478 R15: ffff974a19afcf40
Feb 17 04:15:11 hostA kernel: [472111.801195]  arch_cpu_idle+0x15/0x20
Feb 17 04:15:17 hostA kernel: [472117.824128]  kthreadd+0x29f/0x2f0
Feb 17 04:15:17 hostA kernel: [472118.170708] R10: 00000000000001b8 R11: 0000000000000394 R12: ffff974a1ae218c0
Feb 17 04:15:19 hostA kernel: [472119.834573]  __wake_up_common+0x8d/0x140
Feb 17 04:15:19 hostA kernel: [472120.080314]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Feb 17 04:15:21 hostA kernel: [472122.074906]  sched_autogroup_create_attach+0x9c/0x130
Feb 17 04:15:21 hostA kernel: [472122.271097]  sched_autogroup_create_attach+0x9c/0x130
Feb 17 04:15:22 hostA kernel: [472123.022676] RSP: 002b:00007ffd1ca9dba8 EFLAGS: 00000246 ORIG_RAX: 0000000000000070
Feb 17 04:15:23 hostA kernel: [472123.797140]  __purge_vmap_area_lazy+0x52/0xc0
Feb 17 04:15:23 hostA kernel: [472123.829140]  path_openat+0x617/0x14a0
Feb 17 04:15:34 hostA kernel: [472135.180121]  sys_setsid+0x7c/0x110
Feb 17 04:15:35 hostA kernel: [472135.789089] CR2: 000055e6d3c4d020 CR3: 000000b7e9ec0000 CR4: 00000000003406e0
Feb 17 04:15:39 hostA kernel: [472139.645166]  aperfmperf_snapshot_cpu+0x5c/0x70
Feb 17 04:15:39 hostA kernel: [472139.645250] RSP: 002b:00007f6afcfb2a00 EFLAGS: 00000293 ORIG_RAX: 0000000000000002
Feb 17 04:15:46 hostA kernel: [472147.074297] RBP: ffffb5432a13fca0 R08: 0000000000000000 R09: 0000000000000000
Feb 17 04:15:46 hostA kernel: [472147.074374] R13: 00007fff3b78fff0 R14: 00000000ffffffff R15: 00007fff3b7900e8
Feb 17 04:15:47 hostA kernel: [472148.226666]  cgroup_mkdir+0x34d/0x400
Feb 17 04:15:51 hostA kernel: [472151.797114]  bpf_prog_select_runtime+0xdb/0x100
Feb 17 04:15:51 hostA kernel: [472151.809026] CR2: 00007f85903b5000 CR3: 00000073b9514000 CR4: 00000000003406e0
Feb 17 04:15:51 hostA kernel: [472151.809091] R10: 00007f85854d76a0 R11: 0000000000000206 R12: 0000000000000000
Feb 17 04:15:51 hostA kernel: [472151.829054] Call Trace:
Feb 17 04:15:51 hostA kernel: [472151.829149] RSP: 002b:00007f11bbe3b180 EFLAGS: 00000293 ORIG_RAX: 0000000000000002
Feb 17 04:15:51 hostA kernel: [472152.022876]  rcu_gp_kthread+0x5cc/0x960
Feb 17 04:15:56 hostA kernel: [472157.024479] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 17 04:15:56 hostA kernel: [472157.024555]  cpu_startup_entry+0x73/0x80
Feb 17 04:16:03 hostA kernel: [472163.790702]  ? sync_rcu_exp_select_cpus+0x2ad/0x420
Feb 17 04:16:06 hostA kernel: [472166.778600] CR2: 0000000000a0f210 CR3: 000001fc56d5a000 CR4: 00000000003406e0
Feb 17 04:16:07 hostA kernel: [472167.646744]  arch_freq_prepare_all+0x43/0x90
Feb 17 04:16:13 hostA kernel: [472174.144120]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Feb 17 04:16:13 hostA kernel: [472174.310386] Hardware name: Supermicro Super Server/H11DST-B, BIOS 1.1a 10/04/2018
Feb 17 04:16:13 hostA kernel: [472174.310463] RSP: 0018:ffffb54318b77e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Feb 17 04:16:19 hostA kernel: [472179.734239]  try_to_unmap_one+0x223/0x9b0
Feb 17 04:16:19 hostA kernel: [472179.798944] CR2: 000055e6d4f3a080 CR3: 00000035cae24000 CR4: 00000000003406e0
Feb 17 04:16:19 hostA kernel: [472179.799038] RDX: 000055e6d4f38e10 RSI: 0000000000000000 RDI: 0000000000000001

# uname -a
Linux hostA 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux

# pveversion
pve-manager/5.3-5/97ae681d (running kernel: 4.15.18-9-pve)

# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD EPYC 7601 32-Core Processor
Stepping:              2
CPU MHz:               1199.840
CPU max MHz:           2200.0000
CPU min MHz:           1200.0000
BogoMIPS:              4400.07
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7,64-71
NUMA node1 CPU(s):     8-15,72-79
NUMA node2 CPU(s):     16-23,80-87
NUMA node3 CPU(s):     24-31,88-95
NUMA node4 CPU(s):     32-39,96-103
NUMA node5 CPU(s):     40-47,104-111
NUMA node6 CPU(s):     48-55,112-119
NUMA node7 CPU(s):     56-63,120-127
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

This bug is similar with https://bugzilla.redhat.com/show_bug.cgi?id=619806
and this bug have workaround patch: https://bugzilla.redhat.com/show_bug.cgi?id=681017 :

"Under some circumstances, faulty logic in the system BIOS could report that ASPM (Active State Power Management) was not supported on the system, but leave ASPM enabled on a device. This could lead to AER (Advanced Error Reporting) errors that the kernel was unable to handle. With this update, the kernel proactively disables ASPM on devices when the BIOS reports that ASPM is not supported, safely eliminating the aforementioned issues."

But I suspect this patch was never implemented in vanilla kernel.

Proxmox team may be will you check this and solve?

 

[t.lamprecht]

Linux hostA 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux

This is quite an old kernel release, from one and a half year ago! Please update first and see if you still get the issue with the 4.15.18-53 or newer kernel. Note that also Proxmox VE 6.1 is already out, so maybe worth to update

But I suspect this patch was never implemented in vanilla kernel.

Proxmox team may be will you check this and solve?

Those linked bugs are really old, from 2011. Does the workaround with "pcie_aspm=off", or "pci=noaer" or "pci=nomsi" from your linked reports even help? As else I do not think they are related..

 

[tom]

Please update to latest supported version/kernel and test again, you run a very outdated kernel.

https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_5.x_to_latest_5.4

 

[Alibek]

Unfortunately cluster have 2 nodes with Proxmox 4.x and until this nodes will be upgrade to 5.x I can't update to proxmox 6.x currently (as you know before upgrade to 6.x need update corosync previously).
Upgrade process is going on slowly, because these servers run services that support continuous tcp-connections with hundreds of thousands of devices.

 

[tom]

you requested a fix, but you are unable to apply already available packages with a lot fixes, so it would be hard to help further.

I suggest to upgrade to latest 5.x as a start.

 

[Alibek]

Most servers upgraded, but needs reboot, and two nodes still 4.x (will be updated when possible). But I think error explained in topic start may be present in latest kernels too:

$ for h in {8,9,10,11a,11b,11c,11d,12a,12b,12c,12d}; do echo host$h; ssh root@host$h pveversion; done
host8
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-21-pve)
host9
pve-manager/4.4-22/2728f613 (running kernel: 4.4.95-1-pve)
host10
pve-manager/4.4-24/08ba4d2d (running kernel: 4.4.134-1-pve)
host11a
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-21-pve)
host11b
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
host11c
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-9-pve)
host11d
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-9-pve)
host12a
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
host12b
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
host12c
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)
host12d
pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-24-pve)

 

[tom]

But I think error explained in topic start may be present in latest kernels too

I do not know. In any case, the normal process is to update to latest version and test again. We do not hunt for probably already fixed bugs.

 

[Alibek]

Does the workaround with "pcie_aspm=off", or "pci=noaer" or "pci=nomsi" from your linked reports even help?

Unfortunately I can't check this until next stuck or planned reboot.