Proxmox System Crash

Angelone · Apr 2, 2022

Hi everyone,

I'm experiencing random crashes on my proxmox virtual environment.
This is the stack trace I collected in the syslog :

Bash:

pve kernel: ------------[ cut here ]------------
pve kernel: WARNING: CPU: 10 PID: 0 at kernel/rcu/tree.c:2613 rcu_core+0x5dc/0x660
pve kernel: Modules linked in: udp_diag tcp_diag inet_diag xt_conntrack xt_MASQUERADE xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs binfmt_misc veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd snd_hda_codec_realtek snd_hda_codec_generic amdgpu ledtrig_audio kvm snd_hda_codec_hdmi irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel iommu_v2 gpu_sched snd_hda_intel crypto_simd drm_ttm_helper snd_intel_dspcfg cryptd ttm snd_intel_sdw_acpi snd_hda_codec rapl drm_kms_helper snd_hda_core cec snd_hwdep rc_core pcspkr efi_pstore i2c_algo_bit snd_pcm fb_sys_fops syscopyarea eeepc_wmi sysfillrect snd_timer sysimgblt asus_nb_wmi snd asus_wmi soundcore sparse_keymap video ccp wmi_bmof zfs(PO)
pve kernel:  zunicode(PO) zzstd(O) mac_hid zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbhid hid crc32_pclmul nvme xhci_pci r8169 ahci xhci_pci_renesas gpio_amdpt i2c_piix4 realtek nvme_core xhci_hcd libahci wmi gpio_generic
pve kernel: CPU: 10 PID: 0 Comm: swapper/10 Tainted: P           O      5.13.19-2-pve #1
pve kernel: Hardware name: ASUS System Product Name/TUF GAMING B550M-E, BIOS 2423 08/11/2021
pve kernel: RIP: 0010:rcu_core+0x5dc/0x660
pve kernel: Code: fd ff ff 0f 0b e9 aa fa ff ff fb 66 0f 1f 44 00 00 e9 5e fd ff ff 4c 89 c6 4c 89 f7 e8 bd d1 ae 00 e9 f2 fa ff ff 0f 0b eb cc <0f> 0b e9 35 fd ff ff 0f 0b e9 a9 fa ff ff 0f 0b 4c 89 ff e8 8c 16
pve kernel: RSP: 0018:ffffadc60046cf10 EFLAGS: 00010046
pve kernel: RAX: 0000000000000000 RBX: 0000000000000002 RCX: ffffffff89bd5f21
pve kernel: RDX: 000000000007d8f9 RSI: 0000000000080000 RDI: ffff9d8e4e6adcf0
pve kernel: RBP: ffffadc60046cf70 R08: ffff9d7f9981e7c0 R09: 00000000ffffffff
pve kernel: R10: fffffffffffeecb0 R11: ffff9d7f8c7e0e00 R12: ffff9d8e4e6adc80
pve kernel: R13: ffffadc60046cf28 R14: fffffffffffffffe R15: ffff9d8e4e6adcf0
pve kernel: FS:  0000000000000000(0000) GS:ffff9d8e4e680000(0000) knlGS:0000000000000000
pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
pve kernel: CR2: 00007fcf31a58010 CR3: 0000000799010000 CR4: 0000000000750ee0
pve kernel: PKRU: 55555554
pve kernel: Call Trace:
pve kernel:  <IRQ>
pve kernel:  rcu_core_si+0xe/0x10
pve kernel:  __do_softirq+0xce/0x281
pve kernel:  irq_exit_rcu+0xa2/0xd0
pve kernel:  sysvec_apic_timer_interrupt+0x7c/0x90
pve kernel:  </IRQ>
pve kernel:  asm_sysvec_apic_timer_interrupt+0x12/0x20
pve kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x360
pve kernel: Code: 3d 41 43 0d 77 e8 34 84 7a ff 49 89 c6 0f 1f 44 00 00 31 ff e8 d5 8f 7a ff 80 7d d7 00 0f 85 01 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 ff 0f 88 0d 01 00 00 49 63 cf 4c 2b 75 c8 48 8d 04 49 48 89
pve kernel: RSP: 0018:ffffadc6001b7e68 EFLAGS: 00000246
pve kernel: RAX: ffff9d8e4e6acec0 RBX: 0000000000000003 RCX: 000000000000001f
pve kernel: RDX: 0000000000000000 RSI: 0000000021af2995 RDI: 0000000000000000
pve kernel: RBP: ffffadc6001b7ea0 R08: 000041037d5f1d67 R09: 0000000000000018
pve kernel: R10: 0000000000000e2a R11: ffff9d8e4e6ab884 R12: ffff9d7f8471f400
pve kernel: R13: ffffffff8a45dd80 R14: 000041037d5f1d67 R15: 0000000000000003
pve kernel:  ? cpuidle_enter_state+0xbb/0x360
pve kernel:  cpuidle_enter+0x2e/0x40
pve kernel:  do_idle+0x1ff/0x2a0
pve kernel:  cpu_startup_entry+0x20/0x30
pve kernel:  start_secondary+0x11f/0x160
pve kernel:  secondary_startup_64_no_verify+0xc2/0xcb
pve kernel: ---[ end trace 3702c263834f043b ]---

I'm running Proxmox Virtual Environment 7.1-7 with kernel version 5.13.19-2-pve

My system configuration is:
Motherboard: Asus TUF GAMING B550M-E
CPU: AMD RYZEN 7 5700G
RAM: 4x Corsair Vengeance LPX DDR4-RAM 3600 MHz 16GB (All motherboard slot is fully allocated)
SSD: Samsung MZ-V8V1T0 980

Can anyone help me to solve this issue?
Many thanks in advance

paarak · Apr 2, 2022

Angelone said:

Hi everyone,

I'm experiencing random crashes on my proxmox virtual environment.
This is the stack trace I collected in the syslog :

Bash:

pve kernel: ------------[ cut here ]------------
pve kernel: WARNING: CPU: 10 PID: 0 at kernel/rcu/tree.c:2613 rcu_core+0x5dc/0x660
pve kernel: Modules linked in: udp_diag tcp_diag inet_diag xt_conntrack xt_MASQUERADE xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs binfmt_misc veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common edac_mce_amd kvm_amd snd_hda_codec_realtek snd_hda_codec_generic amdgpu ledtrig_audio kvm snd_hda_codec_hdmi irqbypass crct10dif_pclmul ghash_clmulni_intel aesni_intel iommu_v2 gpu_sched snd_hda_intel crypto_simd drm_ttm_helper snd_intel_dspcfg cryptd ttm snd_intel_sdw_acpi snd_hda_codec rapl drm_kms_helper snd_hda_core cec snd_hwdep rc_core pcspkr efi_pstore i2c_algo_bit snd_pcm fb_sys_fops syscopyarea eeepc_wmi sysfillrect snd_timer sysimgblt asus_nb_wmi snd asus_wmi soundcore sparse_keymap video ccp wmi_bmof zfs(PO)
pve kernel:  zunicode(PO) zzstd(O) mac_hid zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbhid hid crc32_pclmul nvme xhci_pci r8169 ahci xhci_pci_renesas gpio_amdpt i2c_piix4 realtek nvme_core xhci_hcd libahci wmi gpio_generic
pve kernel: CPU: 10 PID: 0 Comm: swapper/10 Tainted: P           O      5.13.19-2-pve #1
pve kernel: Hardware name: ASUS System Product Name/TUF GAMING B550M-E, BIOS 2423 08/11/2021
pve kernel: RIP: 0010:rcu_core+0x5dc/0x660
pve kernel: Code: fd ff ff 0f 0b e9 aa fa ff ff fb 66 0f 1f 44 00 00 e9 5e fd ff ff 4c 89 c6 4c 89 f7 e8 bd d1 ae 00 e9 f2 fa ff ff 0f 0b eb cc <0f> 0b e9 35 fd ff ff 0f 0b e9 a9 fa ff ff 0f 0b 4c 89 ff e8 8c 16
pve kernel: RSP: 0018:ffffadc60046cf10 EFLAGS: 00010046
pve kernel: RAX: 0000000000000000 RBX: 0000000000000002 RCX: ffffffff89bd5f21
pve kernel: RDX: 000000000007d8f9 RSI: 0000000000080000 RDI: ffff9d8e4e6adcf0
pve kernel: RBP: ffffadc60046cf70 R08: ffff9d7f9981e7c0 R09: 00000000ffffffff
pve kernel: R10: fffffffffffeecb0 R11: ffff9d7f8c7e0e00 R12: ffff9d8e4e6adc80
pve kernel: R13: ffffadc60046cf28 R14: fffffffffffffffe R15: ffff9d8e4e6adcf0
pve kernel: FS:  0000000000000000(0000) GS:ffff9d8e4e680000(0000) knlGS:0000000000000000
pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
pve kernel: CR2: 00007fcf31a58010 CR3: 0000000799010000 CR4: 0000000000750ee0
pve kernel: PKRU: 55555554
pve kernel: Call Trace:
pve kernel:  <IRQ>
pve kernel:  rcu_core_si+0xe/0x10
pve kernel:  __do_softirq+0xce/0x281
pve kernel:  irq_exit_rcu+0xa2/0xd0
pve kernel:  sysvec_apic_timer_interrupt+0x7c/0x90
pve kernel:  </IRQ>
pve kernel:  asm_sysvec_apic_timer_interrupt+0x12/0x20
pve kernel: RIP: 0010:cpuidle_enter_state+0xcc/0x360
pve kernel: Code: 3d 41 43 0d 77 e8 34 84 7a ff 49 89 c6 0f 1f 44 00 00 31 ff e8 d5 8f 7a ff 80 7d d7 00 0f 85 01 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 ff 0f 88 0d 01 00 00 49 63 cf 4c 2b 75 c8 48 8d 04 49 48 89
pve kernel: RSP: 0018:ffffadc6001b7e68 EFLAGS: 00000246
pve kernel: RAX: ffff9d8e4e6acec0 RBX: 0000000000000003 RCX: 000000000000001f
pve kernel: RDX: 0000000000000000 RSI: 0000000021af2995 RDI: 0000000000000000
pve kernel: RBP: ffffadc6001b7ea0 R08: 000041037d5f1d67 R09: 0000000000000018
pve kernel: R10: 0000000000000e2a R11: ffff9d8e4e6ab884 R12: ffff9d7f8471f400
pve kernel: R13: ffffffff8a45dd80 R14: 000041037d5f1d67 R15: 0000000000000003
pve kernel:  ? cpuidle_enter_state+0xbb/0x360
pve kernel:  cpuidle_enter+0x2e/0x40
pve kernel:  do_idle+0x1ff/0x2a0
pve kernel:  cpu_startup_entry+0x20/0x30
pve kernel:  start_secondary+0x11f/0x160
pve kernel:  secondary_startup_64_no_verify+0xc2/0xcb
pve kernel: ---[ end trace 3702c263834f043b ]---

I'm running Proxmox Virtual Environment 7.1-7 with kernel version 5.13.19-2-pve

My system configuration is:
Motherboard: Asus TUF GAMING B550M-E
CPU: AMD RYZEN 7 5700G
RAM: 4x Corsair Vengeance LPX DDR4-RAM 3600 MHz 16GB (All motherboard slot is fully allocated)
SSD: Samsung MZ-V8V1T0 980

Can anyone help me to solve this issue?
Many thanks in advance

See if it helps to update the kernel til 5.15

apt update && apt install pve-kernel-5.15

Angelone · Apr 3, 2022

Thanks for the reply, I tried to install the new kernel version but I can't find the package:

Code:

E: Couldn't find any package by glob 'pve-kernel-5.15'
root@pve:~# apt search pve-kernel
Sorting... Done
Full Text Search... Done
pve-firmware/now 3.3-3 all [installed,local]
  Binary firmware code for the pve-kernel

pve-kernel-5.13/now 7.1-5 all [installed,local]
  Latest Proxmox VE Kernel Image

pve-kernel-5.13.19-2-pve/now 5.13.19-4 amd64 [installed,local]
  The Proxmox PVE Kernel Image

pve-kernel-helper/now 7.1-6 all [installed,local]
  Function for various kernel maintenance tasks.

paarak · Apr 7, 2022

Hi, have you updated the repo to no-subscription and commented out # out security updates

1#
nano /etc/apt/sources.list

Code:

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
#deb http://security.debian.org bullseye-security main contrib

2#
nano /etc/apt/sources.list.d/pve-enterprise.list

Code:

#deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

3#
apt update && apt dist-upgrade -y

Neobin · Apr 7, 2022

paarak said:
commented out # out security updates

Why?

paarak · Apr 7, 2022

Neobin said:
Why?

hi if you are using proxmox without a valid subscription and there is an error code 100 in the log you need to update the repo

https://forum.proxmox.com/threads/apt-get-update-fails-with-error-code-100.75162/

fabian · Apr 7, 2022

the debian-security repo is completely separate from ours - it should be enabled no matter which proxmox repository you use.

Neobin · Apr 7, 2022

paarak said:
hi if you are using proxmox without a valid subscription and there is an error code 100 in the log you need to update the repo

https://forum.proxmox.com/threads/apt-get-update-fails-with-error-code-100.75162/

Yes, sure. But my question was, why you are disabling the debian security updates repo.

paarak · Apr 7, 2022

Neobin said:
Yes, sure. But my question was, why you are disabling the debian security updates repo.

you need a valid subscription key to access repository.

Neobin · Apr 7, 2022

paarak said:
you need a valid subscription key to access repository.

Did you even read and understand what I wrote?!
But it's okay, no offense...

datschlatscher · Apr 7, 2022

paarak said:
you need a valid subscription key to access repository.

Yes, you need a valid subscription to access the enterprise subscription repository by Proxmox.
Like Fabian said, the security repository by Debian is in no way connected to the Proxmox repositories. You do not need a subscription key for this repository and really should not disable it.

Angelone · Apr 20, 2022

Hi guys,

I've updated to the pve-kernel-5.15 as you suggested but the crash still happens.
Attached you can find the logs from the starting point until the system stopped to work and I had to reboot it.
Any idea?

Thanks for you support.

Angelone · Apr 26, 2022

Is there anyone that could help me? Thanks

bigmehmet · Apr 27, 2022

Just try to configure your BIOS like stated in the last post
https://forum.proxmox.com/threads/kernel-panic-whole-server-crashes-about-every-day.91803/page-6

Lets see this solves your problem, mine seems stable now..

Angelone · Apr 27, 2022

bigmehmet said:
Just try to configure your BIOS like stated in the last post
https://forum.proxmox.com/threads/kernel-panic-whole-server-crashes-about-every-day.91803/page-6

Lets see this solves your problem, mine seems stable now..

Thank you very much for the info.

I've applyed the setting for SVM and IOMMU but I can't find the one related to "Power idle control".
Do you have any suggestions? My motherboard is an ASUS TUF Gaming B550M-E

Searching the web I saw that someone suggested to disable the "Global C-state control", is it equivalent to the feature "Power support idle control"?

Search

Search

Proxmox System Crash

Angelone

Member

paarak

Member

Angelone

Member

paarak

Member

Neobin

Distinguished Member

paarak

Member

fabian

Proxmox Staff Member

Neobin

Distinguished Member

paarak

Member

Neobin

Distinguished Member

datschlatscher

Proxmox Retired Staff

Angelone

Member

Attachments

Angelone

Member

bigmehmet

New Member

Angelone

Member