Opt-in Linux 6.5 Kernel with ZFS 2.2 for Proxmox VE 8 available on test & no-subscription

Daniel Keller said:
kernel 6.5 works fine except on the one server with Adpatec controller there I get the following error

aacraid: Host adapter abort request.
aacraid: Outstanding commands on (0,1,17,0):
aacraid: Host bus reset request. SCSI hang ?
aacraid 0000:18:00.0: Controller reset type is 3

this is probably the same error as reported here

https://bugzilla.kernel.org/show_bug.cgi?id=217599
Click to expand...

Just out of curiosity what Adaptec card are you using?
 
Hi,
adamb said:
Tested 6.5 on one of my new SuperMicro front ends with 4x Intel Xeon Gold 6448H. VM locks up under load with CPU's stuck. I do run zfs on root with 2 Micron 5400 Pro's.
Click to expand...
could you share the VM configuration? Is there anything in the host's journal/system logs around the time the issue happens? And just to be sure, we're talking about CPU soft lockups inside the guest?
 
Hi,
Ralli said:
Sorry for that simple question: why don't skip the 6.5 und directly go on to 6.6 which presumed would be the next LTS-release?

https://www.heise.de/news/Linux-6-6-bringt-neuen-Scheduler-9350044.html
Click to expand...
because we use the Ubuntu kernel as a base and not directly upstream:
t.lamprecht said:
We recently uploaded a 6.5 kernel into our repositories, it will be used as new default kernel in the next Proxmox VE 8.1 point release (Q4'2023), following our tradition of upgrading the Proxmox VE kernel to match the current Ubuntu version until we reach an (Ubuntu) LTS release.
Click to expand...
 
  • Like
Reactions: Stoiko Ivanov
fiona said:
Hi,

could you share the VM configuration? Is there anything in the host's journal/system logs around the time the issue happens? And just to be sure, we're talking about CPU soft lockups inside the guest?
Click to expand...

Here is the VM config. Its a Debian 12 VM.

Code: 
root@ccsprogmiscrit1:~# cat /etc/pve/nodes/ccsprogmiscrit1/qemu-server/154.conf
agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 62
cpu: host,flags=+pdpe1gb
efidisk0: MissionCrit-Alletra:vm-154-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb,cpu
memory: 768000
meta: creation-qemu=7.2.0,ctime=1690812156
name: Progmaindeb
net0: virtio=2A:B8:4A:01:D8:7A,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: MissionCrit-Alletra:vm-154-disk-1,discard=on,iothread=1,size=100G
scsi1: MissionCrit-Alletra:vm-154-disk-2,backup=0,discard=on,iothread=1,size=6000G
scsihw: virtio-scsi-single
smbios1: uuid=4ac5e432-d411-4d42-b369-2fd98c755a63
sockets: 4
vmgenid: 55cff15b-ea2b-42bf-876e-6a4126786a8b

That is correct soft lockups within the VM itself.

With the both 5.15.x and 6.5.x kernel the host does see some kernel oops related to ZFS. Thats about it though.

5.15.x is rock solid.

Code: 
?zfs
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.533189]  ? init_module_from_file+0x96/0x100
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.535407]  idempotent_init_module+0x11c/0x2b0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.537582]  __x64_sys_finit_module+0x64/0xd0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.539761]  do_syscall_64+0x58/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.541903]  ? syscall_exit_to_user_mode+0x37/0x60
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.544008]  ? do_syscall_64+0x67/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.546137]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.548327] RIP: 0033:0x7fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.550543] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
 73 01 c3 48 8b 0d 37 8f 0d 00 f7 d8 64 89 01 48
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.557762] RSP: 002b:00007ffd5ec52bd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.560297] RAX: ffffffffffffffda RBX: 0000561e33715dd0 RCX: 00007fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.530935]  init_module_from_file+0x96/0x100
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.533189]  ? init_module_from_file+0x96/0x100
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.535407]  idempotent_init_module+0x11c/0x2b0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.537582]  __x64_sys_finit_module+0x64/0xd0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.539761]  do_syscall_64+0x58/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.541903]  ? syscall_exit_to_user_mode+0x37/0x60
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.544008]  ? do_syscall_64+0x67/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.546137]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.548327] RIP: 0033:0x7fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.550543] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
 73 01 c3 48 8b 0d 37 8f 0d 00 f7 d8 64 89 01 48
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.557762] RSP: 002b:00007ffd5ec52bd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.560297] RAX: ffffffffffffffda RBX: 0000561e33715dd0 RCX: 00007fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.562841] RDX: 0000000000000000 RSI: 0000561e32db8260 RDI: 0000000000000004
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.565364] RBP: 0000000000060000 R08: 0000000000000000 R09: 0000561e33715f60
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.567867] R10: 0000000000000004 R11: 0000000000000246 R12: 0000561e32db8260
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.570360] R13: 0000000000000000 R14: 0000561e33715f00 R15: 0000561e33715dd0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.572815]  </TASK>
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.575228] Large kmem_alloc(73728, 0x1000), please file an issue at:
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.575228] https://github.com/openzfs/zfs/issues/new
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.580157] CPU: 133 PID: 2318 Comm: modprobe Tainted: P           O       6.5.3-1-pve #1
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.582670] Hardware name: Supermicro Super Server/X13QEH+, BIOS 1.2 03/23/2023
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.585204] Call Trace:
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.587669]  <TASK>
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.590041]  dump_stack_lvl+0x48/0x70
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.592389]  dump_stack+0x10/0x20
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.594736]  spl_kmem_zalloc+0x11b/0x130 [spl]
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.597108]  zstd_init+0x61/0x1f0 [zfs]
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.599579]  openzfs_init+0x3f/0xbe0 [zfs]
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.602051]  ? __pfx_openzfs_init+0x10/0x10 [zfs]
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.604534]  do_one_initcall+0x5b/0x340
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.606931]  do_init_module+0x68/0x260
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.609309]  load_module+0x213a/0x22a0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.611674]  init_module_from_file+0x96/0x100
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.614051]  ? init_module_from_file+0x96/0x100
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.616446]  idempotent_init_module+0x11c/0x2b0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.618848]  __x64_sys_finit_module+0x64/0xd0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.621228]  do_syscall_64+0x58/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.623598]  ? syscall_exit_to_user_mode+0x37/0x60
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.626008]  ? do_syscall_64+0x67/0x90
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.628403]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.630821] RIP: 0033:0x7fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.633225] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff
 73 01 c3 48 8b 0d 37 8f 0d 00 f7 d8 64 89 01 48
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.640825] RSP: 002b:00007ffd5ec52bd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.643383] RAX: ffffffffffffffda RBX: 0000561e33715dd0 RCX: 00007fa56568ef29
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.645922] RDX: 0000000000000000 RSI: 0000561e32db8260 RDI: 0000000000000004
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.648412] RBP: 0000000000060000 R08: 0000000000000000 R09: 0000561e33715f60
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.650853] R10: 0000000000000004 R11: 0000000000000246 R12: 0000561e32db8260
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.653266] R13: 0000000000000000 R14: 0000561e33715f00 R15: 0000561e33715dd0
Nov  1 14:06:28 ccsprogmiscrit1 kernel: [   25.655670]  </TASK>
 
So far its also looking like KSM is still completely broke in 6.5.

All of this is such a bummer for real enterprise environments.

EDIT:

Might have spoke to soon. We will see were this goes within the next 45 minutes. Went from 0 to 981MB in a matter of a minute or so.

1698946190072.png

EDIT #2:

Yeppers, looking good. Still climbing.

1698948048306.png
 
Last edited:
adamb said:
Here is the VM config. Its a Debian 12 VM.

Code: 
root@ccsprogmiscrit1:~# cat /etc/pve/nodes/ccsprogmiscrit1/qemu-server/154.conf
agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 62
cpu: host,flags=+pdpe1gb
efidisk0: MissionCrit-Alletra:vm-154-disk-0,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb,cpu
memory: 768000
meta: creation-qemu=7.2.0,ctime=1690812156
name: Progmaindeb
net0: virtio=2A:B8:4A:01:D8:7A,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: MissionCrit-Alletra:vm-154-disk-1,discard=on,iothread=1,size=100G
scsi1: MissionCrit-Alletra:vm-154-disk-2,backup=0,discard=on,iothread=1,size=6000G
scsihw: virtio-scsi-single
smbios1: uuid=4ac5e432-d411-4d42-b369-2fd98c755a63
sockets: 4
vmgenid: 55cff15b-ea2b-42bf-876e-6a4126786a8b
Click to expand...
A Quad socket VM with 62 Cores? I think this is not Supportet by KVM. Can you change to 60 or 64 cores and test again?
 
  • Like
Reactions: jsterr
Falk R. said:
A Quad socket VM with 62 Cores? I think this is not Supportet by KVM. Can you change to 60 or 64 cores and test again?
Click to expand...
62 cores is fine (if you have more physical cores than that) but its times 4 sockets, which is 248 virtual cores. What CPUs is @adamb running?
EDIT: @adamb is running 2 sockets of 12-core Xeons, which is only 24 cores (and total 48 hyper-threads). I don't think 248 virtual cores will work.
 
Last edited:
leesteken said:
62 cores is fine (if you have more physical cores than that) but its times 4 sockets, which is 248 virtual cores. What CPUs is @adamb running?
EDIT: @adamb is running 2 sockets of 12-core Xeons, which is only 24 cores (and total 48 hyper-threads). I don't think 248 virtual cores will work.
Click to expand...

I have all kinds of hardware that I test on.

For the KSM issue I have been testing on a HP DL 380 Gen9. That is dual socket.

For the performance issues on the latest 4th Gen Intel CPU's I am testing on a newer Supermicro chassis. Its a quad socket.

https://ark.intel.com/content/www/u...-gold-6448h-processor-60m-cache-2-40-ghz.html

Ive tested this with all kinds of different CPU configurations, fact is there are no issues on 5.15.x, on 6.x any VM is unstable under load with these newer 4th gen intel XEON's
 
Last edited:
leesteken said:
62 cores is fine (if you have more physical cores than that) but its times 4 sockets, which is 248 virtual cores. What CPUs is @adamb running?
EDIT: @adamb is running 2 sockets of 12-core Xeons, which is only 24 cores (and total 48 hyper-threads). I don't think 248 virtual cores will work.
Click to expand...
Only synchron number of cores per Socket are allowed. You cannot split 62 Cores to 4 Sockets.
 
alexskysilk said:
I've never heard this before, and experience shows that isnt actually true- do you have documentation to explain this limitation?
Click to expand...
Hi,
I read a troubleshooting guide from HP on this topic 15 years ago in connection with ESX.
It explained well what problems or side effects can occur if you equip the RAM asynchronously on physical servers and if you mix CPUs with different core numbers. Of course, this also applies to virtual machines.

Have you ever equipped a dual socket server with different CPUs? Probably not.
Intel writes in their guides (unfortunately only available with login and may not be shared freely) when you can mix different CPUs.
1. CPUs must have the same QPI and RAM speed to work together.
2. CPUs must have the same thermal profile (TDP) to work together.
3. the CPUs must have the same number of physical cores to work together.
4. the CPUs must have the same number of logical cores to work together.
5. stepping does not matter.
6. clock speed does not matter.

Points 3 and 4 are of interest to us. The rest only applies to physical servers.
 
leesteken said:
The 62 cores setting is multiplied by the 4 sockets setting, not split: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_cpu
Click to expand...
I almost said it was even worse.
You have 4x 32 cores and give your VM 4x 62 cores.
Yes, the threads from hyperthreading are displayed, but if you want performance, you should never give a VM more cores than are actually available. Now you have to wait for the second thread for every computing operation.
 
Falk R. said:
I almost said it was even worse.
You have 4x 32 cores and give your VM 4x 62 cores.
Yes, the threads from hyperthreading are displayed, but if you want performance, you should never give a VM more cores than are actually available. Now you have to wait for the second thread for every computing operation.
Click to expand...

We have been doing this for a decade with no issues, its not worth the debate.
 
adamb said:
We have been doing this for a decade with no issues, its not worth the debate.
Click to expand...
Possibly yes.
I suspect it's not a bug you're running into.
With ESXi and also with HyperV, the way threads are assigned to VMs has been changed due to the many security vulnerabilities such as spectre, meltdown and so on.
Presumably something has also been changed with KVM and you won't be able to get around the 6.x kernel in the long term.
If you can test this, try a 6.x kernel and only 4x 32cores. If the VM is not currently running permanently at over 50% CPU load, the performance should improve significantly.
 
Falk R. said:
Possibly yes.
I suspect it's not a bug you're running into.
With ESXi and also with HyperV, the way threads are assigned to VMs has been changed due to the many security vulnerabilities such as spectre, meltdown and so on.
Presumably something has also been changed with KVM and you won't be able to get around the 6.x kernel in the long term.
If you can test this, try a 6.x kernel and only 4x 32cores. If the VM is not currently running permanently at over 50% CPU load, the performance should improve significantly.
Click to expand...

I do appreciate the information, but this has nothing to do with that.

This is a Intel Gen 4 XEON issue.

We have Gen2 and Gen3 quad socket setups running 6.x.x kernel with no performance issues.
 
Also noticing this being repeated in the logs on any host I move to the 6.5.x kernel.

[Fri Nov 3 06:51:50 2023] bpfilter: read fail 0
[Fri Nov 3 06:51:50 2023] bpfilter: Loaded bpfilter_umh pid 72492
[Fri Nov 3 06:51:50 2023] bpfilter: read fail 0
[Fri Nov 3 06:51:50 2023] bpfilter: Loaded bpfilter_umh pid 72495
[Fri Nov 3 06:51:50 2023] bpfilter: read fail 0
[Fri Nov 3 06:51:50 2023] bpfilter: Loaded bpfilter_umh pid 72497
[Fri Nov 3 06:51:50 2023] bpfilter: read fail 0

Pretty much scrolling non stop.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back