I have a general purpose, single-node server, which randomly ends up frozen under heavy load, when using kernel 6.2, usually when full backups are running. Woke up to below message this morning.
This server was originally installed with proxmox 7.x, and operated flawlessly. Later upgraded to proxmox 8.0 with kernel 6.2 Random kernel panics started then. Downgrading kernel to 5.15.x seem to solve the issue and had no kernel panics or hard lock-ups under kernel 5.15
Took advantage of a recent update/reboot to switch again to the latest 6.2 kernel, and a day later happened again.
any thoughts?
This server was originally installed with proxmox 7.x, and operated flawlessly. Later upgraded to proxmox 8.0 with kernel 6.2 Random kernel panics started then. Downgrading kernel to 5.15.x seem to solve the issue and had no kernel panics or hard lock-ups under kernel 5.15
Took advantage of a recent update/reboot to switch again to the latest 6.2 kernel, and a day later happened again.
any thoughts?
Code:
2023-10-14T06:11:02.507576-04:00 epyc kernel: [130706.230770] BUG: unable to handle page fault for address: ffffada43b51bf58
2023-10-14T06:11:02.507586-04:00 epyc kernel: [130706.231591] #PF: error_code(0x0011) - permissions violation
2023-10-14T06:11:02.507588-04:00 epyc kernel: [130706.232370] Oops: 0011 [#1] PREEMPT SMP NOPTI
2023-10-14T06:11:02.507588-04:00 epyc kernel: [130706.233128] Hardware name: Supermicro Super Server/H12SSL-NT, BIOS 2.5 09/08/2022
2023-10-14T06:11:02.507589-04:00 epyc kernel: [130706.233513] RIP: 0010:0xffffada43b51bf58
2023-10-14T06:11:02.507589-04:00 epyc kernel: [130706.233876] Code: ff ff a7 61 e7 99 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 eb 00 00 9a ff ff ff ff <70> 02 37 bf ef 55 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2023-10-14T06:11:02.507590-04:00 epyc kernel: [130706.234623] RSP: 0018:ffffada43b522cbd EFLAGS: 00010246
2023-10-14T06:11:02.507591-04:00 epyc kernel: [130706.234991] RAX: 0000000000000001 RBX: ffff9c877021d300 RCX: 0000000000000002
2023-10-14T06:11:02.507591-04:00 epyc kernel: [130706.235750] RBP: ffffada43b51be68 R08: 0000000000000000 R09: 0000000000000000
2023-10-14T06:11:02.507591-04:00 epyc kernel: [130706.236114] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000011
2023-10-14T06:11:02.507592-04:00 epyc kernel: [130706.236843] FS: 00007fd9a5acb6c0(0000) GS:ffff9cc3ceb80000(0000) knlGS:0000000000000000
2023-10-14T06:11:02.507593-04:00 epyc kernel: [130706.237579] CR2: ffffada43b51bf58 CR3: 00000002f90be004 CR4: 0000000000770ee0
2023-10-14T06:11:02.507593-04:00 epyc kernel: [130706.238302] Call Trace:
2023-10-14T06:11:02.507593-04:00 epyc kernel: [130706.238677] Modules linked in: tcp_diag inet_diag veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter scsi_transport_iscsi nf_tables nvme_fabrics sunrpc binfmt_misc bonding nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel ast sha512_ssse3 drm_shmem_helper aesni_intel drm_kms_helper crypto_simd i2c_algo_bit cryptd acpi_ipmi syscopyarea pcspkr rapl ipmi_si wmi_bmof sysfillrect ipmi_devintf sysimgblt ccp k10temp ptdma ipmi_msghandler joydev input_leds mac_hid vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb csiostor rndis_host cdc_ether usbnet mii usbmouse
2023-10-14T06:11:02.507601-04:00 epyc kernel: [130706.242448] CR2: ffffada43b51bf58
2023-10-14T06:11:02.507601-04:00 epyc kernel: [130706.242894] ---[ end trace 0000000000000000 ]---
2023-10-14T06:11:02.507602-04:00 epyc kernel: [130706.374446] RIP: 0010:0xffffada43b51bf58
2023-10-14T06:11:02.507602-04:00 epyc kernel: [130706.375011] Code: ff ff a7 61 e7 99 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 eb 00 00 9a ff ff ff ff <70> 02 37 bf ef 55 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2023-10-14T06:11:02.507603-04:00 epyc kernel: [130706.376163] RSP: 0018:ffffada43b522cbd EFLAGS: 00010246
2023-10-14T06:11:02.507603-04:00 epyc kernel: [130706.376766] RAX: 0000000000000001 RBX: ffff9c877021d300 RCX: 0000000000000002
2023-10-14T06:11:02.507604-04:00 epyc kernel: [130706.377217] RDX: ffff9c88c8666fc0 RSI: ffff9c863dd71088 RDI: 0000000000000011
2023-10-14T06:11:02.507604-04:00 epyc kernel: [130706.377669] RBP: ffffada43b51be68 R08: 0000000000000000 R09: 0000000000000000
2023-10-14T06:11:02.507604-04:00 epyc kernel: [130706.378125] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000011
2023-10-14T06:11:02.507605-04:00 epyc kernel: [130706.378578] R13: 0000000000004000 R14: ffff9c85ac3e7380 R15: 0000000000000011
2023-10-14T06:11:02.507605-04:00 epyc kernel: [130706.379032] FS: 00007fd9a5acb6c0(0000) GS:ffff9cc3ceb80000(0000) knlGS:0000000000000000
2023-10-14T06:11:02.507606-04:00 epyc kernel: [130706.379493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2023-10-14T06:11:02.507606-04:00 epyc kernel: [130706.379958] CR2: ffffada43b51bf58 CR3: 00000002f90be004 CR4: 0000000000770ee0
2023-10-14T06:11:02.507607-04:00 epyc kernel: [130706.380422] PKRU: 55555554
2023-10-14T06:11:02.507607-04:00 epyc kernel: [130706.380886] note: kvm[32877] exited with irqs disabled
Code:
root@epyc:~# pveversion --verbose
proxmox-ve: 8.0.2 (running kernel: 5.15.108-1-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-4
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
proxmox-kernel-6.2.16-14-pve: 6.2.16-14
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1