x86/split lock detection

StreetPiet said:
i'm running a HCI-Cluster with four nodes and from time to time one node freezes completely. I changed the default GRUB file as recommended in the posts above, but i get the message

x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks

when i run journal -b

My boot command-line is:
Command line: initrd=\EFI\proxmox\5.15.30-2-pve\initrd.img-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
Click to expand...

That looks like the changes didn't take. Since you are using zfs, you may need to follow the steps outlined for systemd-boot in the manual [1].

Can you please provide the output of the following commands/content of the following files:
  • proxmox-boot-tool status
  • bootctl
  • /etc/default/grub
  • /etc/kernel/cmdline
Thanks!

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline
 
Last edited:
  • Like
Reactions: StreetPiet
sterzy said:
That looks like the changes didn't take. Since you are using zfs, you may need to follow the steps outlined for systemd-boot in the manual [1].

Can you please provide the output of the following commands/content of the following files:
  • proxmox-boot-tool status
  • bootctl
  • /etc/default/grub
  • /etc/kernel/cmdline
Thanks!
Click to expand...

Here you go...

Code: 
# proxmox-boot-tool  status  -----------------------------------------------------------------

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
0B7E-C78D is configured with: uefi (versions: 5.15.30-2-pve)
0B7F-05A4 is configured with: uefi (versions: 5.15.30-2-pve)

# bootctl ------------------------------------------------------------------------------------
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
System:
     Firmware: UEFI 2.80 (American Megatrends 5.22)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 247.3-7
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

Random Seed:
 Passed to OS: yes
 System Token: set

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0013
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0014
       Status: active, boot-order

# cat /etc/default/grub ----------------------------------------------------------------------

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet split_lock_detect=off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

# cat /etc/kernel/cmdline --------------------------------------------------------------------
root=ZFS=rpool/ROOT/pve-1 boot=zfs

# cat /boot/grub/grub.cfg | grep split -------------------------------------------------------

        linux   /ROOT/pve-1@/boot/vmlinuz-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 ro  root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet split_lock_detect=off
                linux   /ROOT/pve-1@/boot/vmlinuz-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 ro  root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet split_lock_detect=off
 
  • Like
Reactions: StreetPiet
sterzy said:
Yeah this looks like you followed the steps for GRUB instead of systemd-boot. Please follow the instructions for systemd-boot from the manual [1].

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline
Click to expand...
Ahh.. i see!

I added "split_lock_detect=off" to /etc/kernel/cmdline and ran "proxmox-boot-tool refresh" afterwards
after the reboot it worked!

Code: 
kernel: Linux version 5.15.30-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.30-3 (Fri, 22 Apr >
Command line: initrd=\EFI\proxmox\5.15.30-2-pve\initrd.img-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
x86/split lock detection: disabled

Thanks!
 
  • Like
Reactions: sterzy
I also ran into this issue and turned it off in systemd-boot (booting from zfs)

The output from journalctl -b | grep split is a bit puzzling:

root@pve:~# journalctl -b | grep split
Feb 17 21:08:38 pve kernel: Command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: x86/split lock detection: disabled
Feb 17 21:08:38 pve kernel: Kernel command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: Unknown kernel command line parameters "boot=zfs split_lock_detect=off", will be passed to user space.
Feb 17 21:08:38 pve kernel: split_lock_detect=off

It seems off, but i'm not 100% sure?
 
## Oct 27 13:28:00 Prox1 kernel: x86/split lock detection: #AC: CPU 3/KVM/2149 took a split_lock trap at address: 0x7ef1e050

## systemd,1
└─kvm,2110 -id 108 -name NrvAgentDvrWin10,debug-threads=on -no-shutdown -chardevsocket,id=qmp,path=/var/run/qemu-server/108.qmp,server=on,wai
└─{kvm},2149

no i know which vm is causing the split lock, but how do i find out what in the vm is causing this?

i5 13600k, z790 asus mb
kernal 6.2 and 5.15
 
Drallas said:
I also ran into this issue and turned it off in systemd-boot (booting from zfs)

The output from journalctl -b | grep split is a bit puzzling:

root@pve:~# journalctl -b | grep split
Feb 17 21:08:38 pve kernel: Command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: x86/split lock detection: disabled
Feb 17 21:08:38 pve kernel: Kernel command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: Unknown kernel command line parameters "boot=zfs split_lock_detect=off", will be passed to user space.
Feb 17 21:08:38 pve kernel: split_lock_detect=off

It seems off, but i'm not 100% sure?
Click to expand...
Yes, same as me, have you solved?
 
Mikrocok said:
Yes, same as me, have you solved?
Click to expand...
when i see it i say to my self its okay lol, its now causing any hinderance in anything . i just upgraded to 8.1 and latest kernel and i can see my lastest hardware is Woking better and faster than before, in terms of ui , stop start clone etc etc
 
Posting an update here, because this thread is highly ranked when searching for split lock detection:

When this thread was opened, the most recent kernel was 5.15. At that time, as @sterzy pointed out, split lock detection warnings were indeed mostly cosmetic.

However, this has changed in the meantime: For kernels 5.19 and higher (including kernels 6.2, 6.5, 6.8), the host kernel may artificially slow down VM guests when they trigger a split lock. Note, however, that a took a split_lock trap at address warning alone does not mean that split lock detection has a noticeable impact on VM performance. We have set up a dedicated wiki page [1] with instructions how to check if VM performance is indeed affected, and, if that is the case, a summary of options how to avoid that.

[1] https://pve.proxmox.com/wiki/Split_lock_detection
 
  • Like
Reactions: Neobin and sterzy
You must log in or register to reply here.
Top Bottom