x86/split lock detection

StreetPiet said:
i'm running a HCI-Cluster with four nodes and from time to time one node freezes completely. I changed the default GRUB file as recommended in the posts above, but i get the message

x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks

when i run journal -b

My boot command-line is:
Command line: initrd=\EFI\proxmox\5.15.30-2-pve\initrd.img-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
Click to expand...

That looks like the changes didn't take. Since you are using zfs, you may need to follow the steps outlined for systemd-boot in the manual [1].

Can you please provide the output of the following commands/content of the following files:
  • proxmox-boot-tool status
  • bootctl
  • /etc/default/grub
  • /etc/kernel/cmdline
Thanks!

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline
 
Last edited:
  • Like
Reactions: StreetPiet
sterzy said:
That looks like the changes didn't take. Since you are using zfs, you may need to follow the steps outlined for systemd-boot in the manual [1].

Can you please provide the output of the following commands/content of the following files:
  • proxmox-boot-tool status
  • bootctl
  • /etc/default/grub
  • /etc/kernel/cmdline
Thanks!
Click to expand...

Here you go...

Code: 
# proxmox-boot-tool  status  -----------------------------------------------------------------

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
0B7E-C78D is configured with: uefi (versions: 5.15.30-2-pve)
0B7F-05A4 is configured with: uefi (versions: 5.15.30-2-pve)

# bootctl ------------------------------------------------------------------------------------
Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
System:
     Firmware: UEFI 2.80 (American Megatrends 5.22)
  Secure Boot: disabled
   Setup Mode: setup
 Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 247.3-7
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Boot loader sets ESP partition information
WARNING: The boot loader reports different ESP UUID then detected!
          ESP: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

Random Seed:
 Passed to OS: yes
 System Token: set

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0001
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/SYSTEMD/SYSTEMD-BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0013
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
         File: └─/EFI/BOOT/BOOTX64.EFI

        Title: UEFI OS
           ID: 0x0014
       Status: active, boot-order

# cat /etc/default/grub ----------------------------------------------------------------------

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet split_lock_detect=off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

# cat /etc/kernel/cmdline --------------------------------------------------------------------
root=ZFS=rpool/ROOT/pve-1 boot=zfs

# cat /boot/grub/grub.cfg | grep split -------------------------------------------------------

        linux   /ROOT/pve-1@/boot/vmlinuz-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 ro  root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet split_lock_detect=off
                linux   /ROOT/pve-1@/boot/vmlinuz-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 ro  root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet split_lock_detect=off
 
  • Like
Reactions: StreetPiet
sterzy said:
Yeah this looks like you followed the steps for GRUB instead of systemd-boot. Please follow the instructions for systemd-boot from the manual [1].

[1]: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline
Click to expand...
Ahh.. i see!

I added "split_lock_detect=off" to /etc/kernel/cmdline and ran "proxmox-boot-tool refresh" afterwards
after the reboot it worked!

Code: 
kernel: Linux version 5.15.30-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.30-3 (Fri, 22 Apr >
Command line: initrd=\EFI\proxmox\5.15.30-2-pve\initrd.img-5.15.30-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
x86/split lock detection: disabled

Thanks!
 
  • Like
Reactions: sterzy
I also ran into this issue and turned it off in systemd-boot (booting from zfs)

The output from journalctl -b | grep split is a bit puzzling:

root@pve:~# journalctl -b | grep split
Feb 17 21:08:38 pve kernel: Command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: x86/split lock detection: disabled
Feb 17 21:08:38 pve kernel: Kernel command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: Unknown kernel command line parameters "boot=zfs split_lock_detect=off", will be passed to user space.
Feb 17 21:08:38 pve kernel: split_lock_detect=off

It seems off, but i'm not 100% sure?
 
## Oct 27 13:28:00 Prox1 kernel: x86/split lock detection: #AC: CPU 3/KVM/2149 took a split_lock trap at address: 0x7ef1e050

## systemd,1
└─kvm,2110 -id 108 -name NrvAgentDvrWin10,debug-threads=on -no-shutdown -chardevsocket,id=qmp,path=/var/run/qemu-server/108.qmp,server=on,wai
└─{kvm},2149

no i know which vm is causing the split lock, but how do i find out what in the vm is causing this?

i5 13600k, z790 asus mb
kernal 6.2 and 5.15
 
Drallas said:
I also ran into this issue and turned it off in systemd-boot (booting from zfs)

The output from journalctl -b | grep split is a bit puzzling:

root@pve:~# journalctl -b | grep split
Feb 17 21:08:38 pve kernel: Command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: x86/split lock detection: disabled
Feb 17 21:08:38 pve kernel: Kernel command line: initrd=\EFI\proxmox\5.15.85-1-pve\initrd.img-5.15.85-1-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs split_lock_detect=off
Feb 17 21:08:38 pve kernel: Unknown kernel command line parameters "boot=zfs split_lock_detect=off", will be passed to user space.
Feb 17 21:08:38 pve kernel: split_lock_detect=off

It seems off, but i'm not 100% sure?
Click to expand...
Yes, same as me, have you solved?
 
Mikrocok said:
Yes, same as me, have you solved?
Click to expand...
when i see it i say to my self its okay lol, its now causing any hinderance in anything . i just upgraded to 8.1 and latest kernel and i can see my lastest hardware is Woking better and faster than before, in terms of ui , stop start clone etc etc
 
Posting an update here, because this thread is highly ranked when searching for split lock detection:

When this thread was opened, the most recent kernel was 5.15. At that time, as @sterzy pointed out, split lock detection warnings were indeed mostly cosmetic.

However, this has changed in the meantime: For kernels 5.19 and higher (including kernels 6.2, 6.5, 6.8), the host kernel may artificially slow down VM guests when they trigger a split lock. Note, however, that a took a split_lock trap at address warning alone does not mean that split lock detection has a noticeable impact on VM performance. We have set up a dedicated wiki page [1] with instructions how to check if VM performance is indeed affected, and, if that is the case, a summary of options how to avoid that.

[1] https://pve.proxmox.com/wiki/Split_lock_detection
 
  • Like
Reactions: Neobin and sterzy
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom