Opt-in Linux 6.14 Kernel for Proxmox VE 8 available on test & no-subscription

[ragingbull]

Why is the performance loss of AMD-9950X virtualization greatly increased, and the 6.11 kernel does not have this problem

 

[adolfotregosa]

VM still crashing with the newer 6.14 kernel


Apr 11 17:28:32 pve QEMU[92966]: error: kvm run failed Bad address
Apr 11 17:28:32 pve QEMU[92966]: RAX=000018cc0e382840 RBX=0000000000000780 RCX=0000000000000780 RDX=0000000000000780
Apr 11 17:28:32 pve QEMU[92966]: RSI=000018cc0e382840 RDI=00007a71e5d1f000 RBP=00007a71eb854960 RSP=00007a71eb854960
Apr 11 17:28:32 pve QEMU[92966]: R8 =0000000000000780 R9 =0000000000000090 R10=00000000000003c0 R11=0000000000000800
Apr 11 17:28:32 pve QEMU[92966]: R12=0000000000000090 R13=00005597198ef358 R14=00007a71e5d1f000 R15=000018cc0e382840
Apr 11 17:28:32 pve QEMU[92966]: RIP=000055971ff041d0 RFL=00010206 [-----P-] CPL=3 II=0 A20=1 SMM=0 HLT=0
Apr 11 17:28:32 pve QEMU[92966]: ES =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
Apr 11 17:28:32 pve QEMU[92966]: SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Apr 11 17:28:32 pve QEMU[92966]: DS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: FS =0000 00007a71eb8576c0 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: GS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: LDT=0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: TR =0040 fffffe7be1176000 00004087 00008b00 DPL=0 TSS64-busy
Apr 11 17:28:32 pve QEMU[92966]: GDT= fffffe7be1174000 0000007f
Apr 11 17:28:32 pve QEMU[92966]: IDT= fffffe0000000000 00000fff
Apr 11 17:28:32 pve QEMU[92966]: CR0=80050033 CR2=00007a71e5d1f000 CR3=000000011ec9a006 CR4=00772ef0
Apr 11 17:28:32 pve QEMU[92966]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Apr 11 17:28:32 pve QEMU[92966]: DR6=00000000ffff0ff0 DR7=0000000000000400
Apr 11 17:28:32 pve QEMU[92966]: EFER=0000000000000d01
Apr 11 17:28:32 pve QEMU[92966]: Code=cc cc cc cc 55 48 89 e5 48 89 f8 48 63 ca 48 89 f7 48 89 c6 <f3> a4 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc 55 48 89 e5 89 55 fc f3 0f 6f 07 f3 0f 6f
Apr 11 17:28:32 pve QEMU[92966]: RAX=000018cc0e303040 RBX=0000000000000780 RCX=0000000000000780 RDX=0000000000000780
Apr 11 17:28:32 pve QEMU[92966]: RSI=000018cc0e303040 RDI=00007a71e5c97000 RBP=00007a72793fa960 RSP=00007a72793fa960
Apr 11 17:28:32 pve QEMU[92966]: R8 =0000000000000780 R9 =0000000000000110 R10=00000000000003c0 R11=0000000000000800
Apr 11 17:28:32 pve QEMU[92966]: R12=0000000000000110 R13=00005597198ef358 R14=00007a71e5c97000 R15=000018cc0e303040
Apr 11 17:28:32 pve QEMU[92966]: RIP=000055971ff041d0 RFL=00010202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
Apr 11 17:28:32 pve QEMU[92966]: ES =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
Apr 11 17:28:32 pve QEMU[92966]: SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Apr 11 17:28:32 pve QEMU[92966]: DS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: FS =0000 00007a72793fd6c0 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: GS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: LDT=0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: TR =0040 fffffe698ad4f000 00004087 00008b00 DPL=0 TSS64-busy
Apr 11 17:28:32 pve QEMU[92966]: GDT= fffffe698ad4d000 0000007f
Apr 11 17:28:32 pve QEMU[92966]: IDT= fffffe0000000000 00000fff
Apr 11 17:28:32 pve QEMU[92966]: CR0=80050033 CR2=00007a71e5c97000 CR3=000000011ec9a003 CR4=00772ef0
Apr 11 17:28:32 pve QEMU[92966]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Apr 11 17:28:32 pve QEMU[92966]: DR6=00000000ffff0ff0 DR7=0000000000000400
Apr 11 17:28:32 pve QEMU[92966]: EFER=0000000000000d01
Apr 11 17:28:32 pve QEMU[92966]: Code=cc cc cc cc 55 48 89 e5 48 89 f8 48 63 ca 48 89 f7 48 89 c6 <f3> a4 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc 55 48 89 e5 89 55 fc f3 0f 6f 07 f3 0f 6f
Apr 11 17:28:32 pve QEMU[92966]: RAX=000018cc0e283840 RBX=0000000000000780 RCX=0000000000000780 RDX=0000000000000780
Apr 11 17:28:32 pve QEMU[92966]: RSI=000018cc0e283840 RDI=00007a71e5c0f000 RBP=00007a71ed884960 RSP=00007a71ed884960
Apr 11 17:28:32 pve QEMU[92966]: R8 =0000000000000780 R9 =0000000000000110 R10=00000000000003c0 R11=0000000000000800
Apr 11 17:28:32 pve QEMU[92966]: R12=0000000000000110 R13=00005597198ef358 R14=00007a71e5c0f000 R15=000018cc0e283840
Apr 11 17:28:32 pve QEMU[92966]: RIP=000055971ff041d0 RFL=00010202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
Apr 11 17:28:32 pve QEMU[92966]: ES =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
Apr 11 17:28:32 pve QEMU[92966]: SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Apr 11 17:28:32 pve QEMU[92966]: DS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: FS =0000 00007a71ed8876c0 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: GS =0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: LDT=0000 0000000000000000 ffffffff 00c00000
Apr 11 17:28:32 pve QEMU[92966]: TR =0040 fffffe1d5342d000 00004087 00008b00 DPL=0 TSS64-busy
Apr 11 17:28:32 pve QEMU[92966]: GDT= fffffe1d5342b000 0000007f
Apr 11 17:28:32 pve QEMU[92966]: IDT= fffffe0000000000 00000fff
Apr 11 17:28:32 pve QEMU[92966]: CR0=80050033 CR2=00007a71e5c0f000 CR3=000000011ec9a004 CR4=00772ef0
Apr 11 17:28:32 pve QEMU[92966]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Apr 11 17:28:32 pve QEMU[92966]: DR6=00000000ffff0ff0 DR7=0000000000000400
Apr 11 17:28:32 pve QEMU[92966]: EFER=0000000000000d01
Apr 11 17:28:32 pve QEMU[92966]: Code=cc cc cc cc 55 48 89 e5 48 89 f8 48 63 ca 48 89 f7 48 89 c6 <f3> a4 5d c3 cc cc cc cc cc cc cc cc cc cc cc cc 55 48 89 e5 89 55 fc f3 0f 6f 07 f3 0f 6f

 

[danmcq]

I've tested the 6.14.0-2-pve kernel on my AMD EPYC 9754 system. The problem still exists on my system, when the ae4dma module is loaded.

Same here. I'm still unable to boot my Epyc 9554 system on 6.14.0-2-pve without blacklisting the ae4dma driver.

 

[Hanhi]

https://bugzilla.proxmox.com/show_bug.cgi?id=6273

A

Thread 'e1000 driver hang'

Sep 23, 2019

In the past week we are seeing random e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang failuresacross all our nodes, even different hardware hosts. Must do a reset of the host.

There are lots of references to this issue going back 5+ years. Was there a driver change with the latest updates? We've run years with this hardware without issue. Now just this week its popping up all over.

Kernel Version Linux 5.0.21-2-pve #1 SMP PVE 5.0.21-3 (Thu, 05 Sep 2019 13:56:01 +0200)
PVE Manager Version pve-manager/6.0-7/2898402

Sep 22 20:03:08 vmhost03 kernel: [154458.471981] e1000e...

• alatteri
• Replies: 258
• Forum: Proxmox VE: Networking and Firewall

• 

E1000 (Intel I219-LM) hang exists also on 6.14.0-2 - reminder to all those who wished that it would fix it. Host collapses in hours now.

 

[zenowl77]

https://github.com/GreenDamTan/vgpu_unlock-rs/commit/a651cf35c0097426c369c2783c749b2ed28bce39
try to use A5500 device id in pascal card

Thank you, much appreciated

 

[zenowl77]

Compiled from zfs source and all running good with last kernel. great success
View attachment 84822

Are there dangers to doing this? I really would like to get direct i/o and fast dedup working but i thought there were custom aspects to the pve variant of zfs and we shouldn’t install from zfs source?

 

[proxynice]

Why is the performance loss of AMD-9950X virtualization greatly increased, and the 6.11 kernel does not have this problem

I'm having the same performance problem with my AMD 9950X3D in 6.14. Going back to 6.11 the performance is greatly increased again: In a Windows 11 VM with 8 cores/16 threads I get 61 FPS in X-Plane, while I only get 50 FPS in 6.14. In Cinebench R23 I get around 31.100 pts in 6.11 and only 28.900 pts in 6.14. I've checked that there is no thermal throttling. Going back and forth between the two kernels, the results are reliable repeatable.

The weird thing is that the clocks in my Windows VM in 6.14 are even higher under load (and in idle) than in 6.11, so theoretically the performance should be increased, not decreased. Maybe this is a timing problem with all the changes to the AMD P-State Driver? Sadly, I don't know how to get to the bottom of the problem.

Edit: Manually pinning the CPU Affinity of the 8 cores/16 threads of the VM to CCD0 (0-7,16-23) in 6.14 brings back the performance of 6.11 - at least in X-Plane (61 fps). However, in this scenario I only get 22.000 pts in Cinebench R23 instead of 31.100 pts. No CPU pinning for any scenario (gaming/benchmarking) was needed in 6.11 in order to get the best results. I wonder what changed under the hood.

 

[jauling]

FYI: There's a newer proxmox-kernel in version 6.14.0-2-pve available on the pvetest repo, it should address the issues with AMD EPYC Genoa and the ae4dma module and also reduce the log spam for some Intel N CPU based system.

nice, yeah it looks like I guess I don't have ECC errors

# dmesg | egrep -i 'edac|igen'
[    0.957784] EDAC MC: Ver: 3.0.0
[    3.629302] caller igen6_probe+0x1bc/0x8e0 [igen6_edac] mapping multiple BARs
[    3.629592] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (POLLED)
[    3.629636] EDAC igen6: v2.5.1

Still higher idle than latest 6.8.12 though