My Proxmox server keeps crashing randomly. At work we have similar servers with each running Proxmox VE. It runs perfect on the other 4 but somehow it doesnt on this one. I dont know how to describe this problem since there is no indication of anything being wrong.
Heres my journalctl, according to PRTG ,our monitoring system, the crash must have occured somewhere around 00:00 till 00:07
Dec 11 23:03:16 PD-SE-104 smartd[888]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 77 to 76
Dec 11 23:08:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:11:28 PD-SE-104 pveproxy[279672]: worker exit
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: worker 279672 finished
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: worker 291409 started
Dec 11 23:16:01 PD-SE-104 pveproxy[279673]: worker exit
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: worker 279673 finished
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: worker 293038 started
Dec 11 23:17:01 PD-SE-104 CRON[293401]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 11 23:17:01 PD-SE-104 CRON[293402]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Dec 11 23:17:01 PD-SE-104 CRON[293401]: pam_unix(cron:session): session closed for user root
Dec 11 23:17:23 PD-SE-104 pveproxy[282983]: worker exit
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: worker 282983 finished
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: worker 293539 started
Dec 11 23:23:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:33:16 PD-SE-104 smartd[888]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 76 to 77
Dec 11 23:37:59 PD-SE-104 pveproxy[293539]: worker exit
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: worker 293539 finished
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: worker 300983 started
Dec 11 23:38:07 PD-SE-104 pvedaemon[23279]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:47:02 PD-SE-104 pveproxy[291409]: worker exit
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: worker 291409 finished
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: worker 304236 started
Dec 11 23:53:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 12 00:00:33 PD-SE-104 pveproxy[300983]: worker exit
Dec 12 00:00:33 PD-SE-104 systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Dec 12 00:00:33 PD-SE-104 systemd[1]: Starting logrotate.service - Rotate log files...
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: worker 300983 finished
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: worker 309138 started
Dec 12 00:00:33 PD-SE-104 systemd[1]: logrotate.service: Deactivated successfully.
Dec 12 00:00:33 PD-SE-104 systemd[1]: Finished logrotate.service - Rotate log files.
Dec 12 00:00:33 PD-SE-104 systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Dec 12 00:00:33 PD-SE-104 systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
-- Boot b2b7824a62c5469aa0725474f921c092 --
Dec 12 00:10:58 PD-SE-104 kernel: Linux version 6.8.12-5-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PR>
Dec 12 00:10:58 PD-SE-104 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-5-pve root=/dev/mapper/pve-root ro quiet
Dec 12 00:10:58 PD-SE-104 kernel: KERNEL supported cpus:
Dec 12 00:10:58 PD-SE-104 kernel: Intel GenuineIntel
Dec 12 00:10:58 PD-SE-104 kernel: AMD AuthenticAMD
Dec 12 00:10:58 PD-SE-104 kernel: Hygon HygonGenuine
Dec 12 00:10:58 PD-SE-104 kernel: Centaur CentaurHauls
Dec 12 00:10:58 PD-SE-104 kernel: zhaoxin Shanghai
Dec 12 00:10:58 PD-SE-104 kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-provided physical RAM map:
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000074519fff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007451a000-0x0000000078219fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007821a000-0x00000000784adfff] ACPI data
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000784ae000-0x000000007871efff] ACPI NVS
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007871f000-0x0000000079f1efff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000079f1f000-0x0000000079ffefff] type 20
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000079fff000-0x0000000079ffffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007a000000-0x000000007dffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007ee00000-0x000000007fffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000107fffffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: NX (Execute Disable) protection: active
Dec 12 00:10:58 PD-SE-104 kernel: APIC: Static calls initialized
Dec 12 00:10:58 PD-SE-104 kernel: efi: EFI v2.8 by American Megatrends
Dec 12 00:10:58 PD-SE-104 kernel: efi: ACPI=0x784ad000 ACPI 2.0=0x784ad014 TPMFinalLog=0x7866c000 SMBIOS=0x79a03000 SMBIOS 3.0=0x79a02000 MEMATTR=0x701a6318 >
Dec 12 00:10:58 PD-SE-104 kernel: efi: Remove mem99: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: e820: remove [mem 0xc0000000-0xcfffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem100: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem101: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem102: MMIO range=[0xfed00000-0xfed00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem104: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Remove mem105: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Heres my journalctl, according to PRTG ,our monitoring system, the crash must have occured somewhere around 00:00 till 00:07
Dec 11 23:03:16 PD-SE-104 smartd[888]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 77 to 76
Dec 11 23:08:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:11:28 PD-SE-104 pveproxy[279672]: worker exit
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: worker 279672 finished
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:11:28 PD-SE-104 pveproxy[1287]: worker 291409 started
Dec 11 23:16:01 PD-SE-104 pveproxy[279673]: worker exit
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: worker 279673 finished
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:16:01 PD-SE-104 pveproxy[1287]: worker 293038 started
Dec 11 23:17:01 PD-SE-104 CRON[293401]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 11 23:17:01 PD-SE-104 CRON[293402]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Dec 11 23:17:01 PD-SE-104 CRON[293401]: pam_unix(cron:session): session closed for user root
Dec 11 23:17:23 PD-SE-104 pveproxy[282983]: worker exit
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: worker 282983 finished
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:17:24 PD-SE-104 pveproxy[1287]: worker 293539 started
Dec 11 23:23:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:33:16 PD-SE-104 smartd[888]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 76 to 77
Dec 11 23:37:59 PD-SE-104 pveproxy[293539]: worker exit
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: worker 293539 finished
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:37:59 PD-SE-104 pveproxy[1287]: worker 300983 started
Dec 11 23:38:07 PD-SE-104 pvedaemon[23279]: <root@pam> successful auth for user 'root@pam'
Dec 11 23:47:02 PD-SE-104 pveproxy[291409]: worker exit
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: worker 291409 finished
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 11 23:47:02 PD-SE-104 pveproxy[1287]: worker 304236 started
Dec 11 23:53:07 PD-SE-104 pvedaemon[133941]: <root@pam> successful auth for user 'root@pam'
Dec 12 00:00:33 PD-SE-104 pveproxy[300983]: worker exit
Dec 12 00:00:33 PD-SE-104 systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Dec 12 00:00:33 PD-SE-104 systemd[1]: Starting logrotate.service - Rotate log files...
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: worker 300983 finished
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: starting 1 worker(s)
Dec 12 00:00:33 PD-SE-104 pveproxy[1287]: worker 309138 started
Dec 12 00:00:33 PD-SE-104 systemd[1]: logrotate.service: Deactivated successfully.
Dec 12 00:00:33 PD-SE-104 systemd[1]: Finished logrotate.service - Rotate log files.
Dec 12 00:00:33 PD-SE-104 systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Dec 12 00:00:33 PD-SE-104 systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
-- Boot b2b7824a62c5469aa0725474f921c092 --
Dec 12 00:10:58 PD-SE-104 kernel: Linux version 6.8.12-5-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PR>
Dec 12 00:10:58 PD-SE-104 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-5-pve root=/dev/mapper/pve-root ro quiet
Dec 12 00:10:58 PD-SE-104 kernel: KERNEL supported cpus:
Dec 12 00:10:58 PD-SE-104 kernel: Intel GenuineIntel
Dec 12 00:10:58 PD-SE-104 kernel: AMD AuthenticAMD
Dec 12 00:10:58 PD-SE-104 kernel: Hygon HygonGenuine
Dec 12 00:10:58 PD-SE-104 kernel: Centaur CentaurHauls
Dec 12 00:10:58 PD-SE-104 kernel: zhaoxin Shanghai
Dec 12 00:10:58 PD-SE-104 kernel: x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-provided physical RAM map:
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000000009e000-0x000000000009efff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000074519fff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007451a000-0x0000000078219fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007821a000-0x00000000784adfff] ACPI data
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000784ae000-0x000000007871efff] ACPI NVS
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007871f000-0x0000000079f1efff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000079f1f000-0x0000000079ffefff] type 20
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000079fff000-0x0000000079ffffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007a000000-0x000000007dffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x000000007ee00000-0x000000007fffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000c0000000-0x00000000cfffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fed20000-0x00000000fed7ffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000107fffffff] usable
Dec 12 00:10:58 PD-SE-104 kernel: NX (Execute Disable) protection: active
Dec 12 00:10:58 PD-SE-104 kernel: APIC: Static calls initialized
Dec 12 00:10:58 PD-SE-104 kernel: efi: EFI v2.8 by American Megatrends
Dec 12 00:10:58 PD-SE-104 kernel: efi: ACPI=0x784ad000 ACPI 2.0=0x784ad014 TPMFinalLog=0x7866c000 SMBIOS=0x79a03000 SMBIOS 3.0=0x79a02000 MEMATTR=0x701a6318 >
Dec 12 00:10:58 PD-SE-104 kernel: efi: Remove mem99: MMIO range=[0xc0000000-0xcfffffff] (256MB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: e820: remove [mem 0xc0000000-0xcfffffff] reserved
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem100: MMIO range=[0xfe000000-0xfe010fff] (68KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem101: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem102: MMIO range=[0xfed00000-0xfed00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Not removing mem104: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Dec 12 00:10:58 PD-SE-104 kernel: efi: Remove mem105: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
