Proxmox VE 6.4-8 crashes several times a day

[Whalefin]

Hello!

I have a problem since 3 or 4 days. I reinstalled Proxmox today as my last straw but it still crashed twice just today and I have absolutely no idea why. It ran absolutely fine since January this year. When it crashes I have to manually reboot the NUC because neither the Web Interface is reachable, nor do I have access via ssh. Its completely unresponsive every time.

My Proxmox VE 6.4-8 has one VM running (HomeAssistant). This time it crashed at 14:26. And at 14:56 I manually restarted it. I cant find something in the logs:

auth.log

Jun 23 12:17:01 proxmox CRON[29825]: pam_unix(cron:session): session opened for user root by (uid=0)
Jun 23 12:17:01 proxmox CRON[29825]: pam_unix(cron:session): session closed for user root
Jun 23 13:17:01 proxmox CRON[5893]: pam_unix(cron:session): session opened for user root by (uid=0)
Jun 23 13:17:01 proxmox CRON[5893]: pam_unix(cron:session): session closed for user root
Jun 23 14:17:01 proxmox CRON[14360]: pam_unix(cron:session): session opened for user root by (uid=0)
Jun 23 14:17:01 proxmox CRON[14360]: pam_unix(cron:session): session closed for user root
Jun 23 14:56:31 proxmox systemd-logind[640]: New seat seat0.

deamon.log

Jun 23 14:24:02 proxmox systemd[1]: pvesr.service: Succeeded.
Jun 23 14:24:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Jun 23 14:25:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Jun 23 14:25:02 proxmox systemd[1]: pvesr.service: Succeeded.
Jun 23 14:25:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Jun 23 14:26:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Jun 23 14:26:02 proxmox systemd[1]: pvesr.service: Succeeded.
Jun 23 14:26:02 proxmox systemd[1]: Started Proxmox VE replication runner.
                                                                                                                                                                                                                       Jun 23 14:56:31 proxmox systemd-modules-load[350]: Inserted module 'iscsi_tcp'
Jun 23 14:56:31 proxmox dmeventd[358]: dmeventd ready for processing.

kern.log

Jun 23 10:15:07 proxmox kernel: [    9.402309] vmbr0: port 1(enp3s0) entered blocking state
Jun 23 11:45:01 proxmox kernel: [ 5403.444690] device tap100i0 entered promiscuous mode
Jun 23 11:45:01 proxmox kernel: [ 5403.465700] vmbr0: port 2(tap100i0) entered blocking state
Jun 23 11:45:01 proxmox kernel: [ 5403.465705] vmbr0: port 2(tap100i0) entered disabled state
Jun 23 11:45:01 proxmox kernel: [ 5403.465877] vmbr0: port 2(tap100i0) entered blocking state
Jun 23 11:45:01 proxmox kernel: [ 5403.465880] vmbr0: port 2(tap100i0) entered forwarding state
Jun 23 11:45:09 proxmox kernel: [ 5411.278698] usb 1-3: reset full-speed USB device number 2 using xhci_hcd
Jun 23 11:46:24 proxmox kernel: [ 5485.839608] perf: interrupt took too long (4925 > 4923), lowering kernel.perf_event_max_sample_rate to 40500
Jun 23 12:52:12 proxmox kernel: [ 9434.651133] perf: interrupt took too long (6160 > 6156), lowering kernel.perf_event_max_sample_rate to 32250
Jun 23 14:56:31 proxmox kernel: [    0.000000] Linux version 5.4.119-1-pve (build@proxmox) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.119-1 (Tue, 01 Jun 2021 15:32:00 +0200) ()
Jun 23 14:56:31 proxmox kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.119-1-pve root=/dev/mapper/pve-root ro quiet

syslog (Cant copy the loads of "nulnulnul" stuff so I just written it into the log for you to see)

Jun 23 14:25:02 proxmox systemd[1]: pvesr.service: Succeeded.
Jun 23 14:25:02 proxmox systemd[1]: Started Proxmox VE replication runner.
Jun 23 14:26:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Jun 23 14:26:02 proxmox systemd[1]: pvesr.service: Succeeded.
Jun 23 14:26:02 proxmox systemd[1]: Started Proxmox VE replication runner.
NULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNULNUL                                                                                                                                                                                                 Jun 23 14:56:31 proxmox kernel: [    0.000000] Linux version 5.4.119-1-pve (build@proxmox) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.119-1 (Tue, 01 Jun 2021 15:32:00 +0200) ()
Jun 23 14:56:31 proxmox kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.119-1-pve root=/dev/mapper/pve-root ro quiet
Jun 23 14:56:31 proxmox kernel: [    0.000000] KERNEL supported cpus:
Jun 23 14:56:31 proxmox kernel: [    0.000000]   Intel GenuineIntel
Jun 23 14:56:31 proxmox kernel: [    0.000000]   AMD AuthenticAMD
Jun 23 14:56:31 proxmox kernel: [    0.000000]   Hygon HygonGenuine
Jun 23 14:56:31 proxmox kernel: [    0.000000]   Centaur CentaurHauls
Jun 23 14:56:31 proxmox kernel: [    0.000000]   zhaoxin   Shanghai

 

[mira]

Please provide the complete syslog entries from 14:00 until 15:15. These should be in /var/log/syslog.2.gz by now.

 

[Whalefin]

Yes sure!

Sorry I have to attach it as a file because I get an error when trying to post it (too long). Also theres only a syslog and a syslog.1, I guess thats the right one then?

 

[mira]

Thank you for the syslog. It seems you've customized the syslog config as it contains multiple days instead of just one.
And the order of some of the log entries seems strange as well.

As the reboots are happening without any warning, I'd assume it is a hardware issue. Perhaps the power supply or memory.

Jun 24 13:52:50 proxmox kernel: [28172.926028] BUG: unable to handle page fault for address: ffffffff8d482434
Jun 24 13:52:50 proxmox kernel: [28172.926090] #PF: supervisor write access in kernel mode
Jun 24 13:52:50 proxmox kernel: [28172.926124] #PF: error_code(0x0002) - not-present page
Jun 24 13:52:50 proxmox kernel: [28172.926158] PGD 49e0e067 P4D 49e0e067 PUD 49e0f063 PMD 0
Jun 24 13:52:50 proxmox kernel: [28172.926196] Oops: 0002 [#1] SMP NOPTI
Jun 24 13:52:50 proxmox kernel: [28172.926222] CPU: 1 PID: 16352 Comm: kvm Tainted: P           O      5.4.119-1-pve #1
Jun 24 13:52:50 proxmox kernel: [28172.926272] Hardware name: Intel(R) Client Systems NUC6CAYH/NUC6CAYB, BIOS AYAPLCEL.86A.0064.2019.0910.1422 09/10/2019
Jun 24 13:52:50 proxmox kernel: [28172.926343] RIP: 0010:poll_select_finish+0x8e/0x210
Jun 24 13:52:50 proxmox kernel: [28172.926376] Code: 00 00 04 0f 85 ae 00 00 00 49 83 3c 24 00 75 0c 49 83 7c 24 08 00 0f 84 a7 00 00 00 48 8d 7d b8 e8 e7 89 e4 ff 49 8b 54 24 08 <09> 8b 34 24 48 8d 7d c8 48 2b 55 c0 48 2b 75 b8 e8 2d 1b e4 ff 48
Jun 24 13:52:50 proxmox kernel: [28172.926489] RSP: 0018:ffffa39600547e78 EFLAGS: 00010202
Jun 24 13:52:50 proxmox kernel: [28172.926524] RAX: 0000000016dde4d1 RBX: 0000000000000000 RCX: 0000000000000018
Jun 24 13:52:50 proxmox kernel: [28172.926569] RDX: 0000000016dd2882 RSI: 0000000000006e0d RDI: 000000002ada6bb6
Jun 24 13:52:50 proxmox kernel: [28172.926614] RBP: ffffa39600547ec0 R08: 00274c0d55f78f2c R09: ffff8e333ffd2d00
Jun 24 13:52:50 proxmox kernel: [28172.926660] R10: 0000000000000000 R11: ffff8e333ffd2000 R12: ffffa39600547ee8
Jun 24 13:52:50 proxmox kernel: [28172.926705] R13: 00007ffc53ce3c50 R14: 0000000000000002 R15: 000000000000004d
Jun 24 13:52:50 proxmox kernel: [28172.926751] FS:  00007fc864d75840(0000) GS:ffff8e3337a80000(0000) knlGS:0000000000000000
Jun 24 13:52:50 proxmox kernel: [28172.926802] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 24 13:52:50 proxmox kernel: [28172.926838] CR2: ffffffff8d482434 CR3: 00000001026ae000 CR4: 00000000003426e0
Jun 24 13:52:50 proxmox kernel: [28172.926883] Call Trace:
Jun 24 13:52:50 proxmox kernel: [28172.926906]  __x64_sys_ppoll+0xc0/0xf0
Jun 24 13:52:50 proxmox kernel: [28172.926935]  do_syscall_64+0x57/0x190
Jun 24 13:52:50 proxmox kernel: [28172.926962]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jun 24 13:52:50 proxmox kernel: [28172.926996] RIP: 0033:0x7fc871942916
Jun 24 13:52:50 proxmox kernel: [28172.927021] Code: 7c 24 08 e8 5c 7e 01 00 41 b8 08 00 00 00 4c 8b 54 24 18 48 89 da 41 89 c1 48 8b 74 24 10 48 8b 7c 24 08 b8 0f 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 28 44 89 cf 89 44 24 08 e8 86 7e 01 00 8b 44
Jun 24 13:52:50 proxmox kernel: [28172.927133] RSP: 002b:00007ffc53ce3c30 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
Jun 24 13:52:50 proxmox kernel: [28172.927181] RAX: ffffffffffffffda RBX: 00007ffc53ce3c50 RCX: 00007fc871942916
Jun 24 13:52:50 proxmox kernel: [28172.927225] RDX: 00007ffc53ce3c50 RSI: 000000000000004d RDI: 000055f58a64c800
Jun 24 13:52:50 proxmox kernel: [28172.927270] RBP: 00007ffc53ce3cc0 R08: 0000000000000008 R09: 0000000000000000
Jun 24 13:52:50 proxmox kernel: [28172.927315] R10: 0000000000000000 R11: 0000000000000293 R12: 000055f589196c00
Jun 24 13:52:50 proxmox kernel: [28172.927360] R13: 000055f589196c00 R14: 00007ffc53ce3cbc R15: 0000000000000000
Jun 24 13:52:50 proxmox kernel: [28172.927405] Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter softdog nfnetlink_log nfnetlink intel_rapl_msr snd_hda_codec_hdmi intel_rapl_common mei_hdcp intel_telemetry_pltdrv intel_punit_ipc snd_hda_codec_realtek intel_telemetry_core intel_pmc_ipc snd_hda_codec_generic snd_sof_pci snd_sof_intel_hda_common snd_soc_hdac_hda x86_pkg_temp_thermal intel_powerclamp coretemp snd_sof_intel_hda snd_sof_intel_byt kvm_intel snd_sof_intel_ipc snd_sof snd_sof_xtensa_dsp snd_hda_ext_core kvm snd_soc_acpi_intel_match snd_soc_acpi ledtrig_audio irqbypass snd_soc_core crct10dif_pclmul crc32_pclmul snd_compress ghash_clmulni_intel ac97_bus snd_pcm_dmaengine iwlmvm mac80211 snd_hda_intel libarc4 btusb btrtl i915 btbcm snd_intel_dspcfg aesni_intel btintel crypto_simd snd_hda_codec cryptd bluetooth glue_helper snd_hda_core iwlwifi rapl drm_kms_helper snd_hwdep intel_cstate input_leds ecdh_generic snd_pcm drm
Jun 24 13:52:50 proxmox kernel: [28172.927448]  wmi_bmof serio_raw i2c_algo_bit cdc_acm fb_sys_fops ecc 8250_dw mei_me syscopyarea pcspkr snd_timer rtsx_pci_ms sysfillrect sysimgblt cfg80211 snd memstick mei intel_xhci_usb_role_switch soundcore roles mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c spi_pxa2xx_platform dw_dmac dw_dmac_core rtsx_pci_sdmmc psmouse i2c_i801 lpc_ich intel_lpss_pci xhci_pci intel_lpss idma64 xhci_hcd virt_dma r8169 realtek ahci rtsx_pci libahci wmi video pinctrl_broxton pinctrl_intel
Jun 24 13:52:50 proxmox kernel: [28172.928400] CR2: ffffffff8d482434
Jun 24 13:52:50 proxmox kernel: [28172.928425] ---[ end trace 0cb3c79b3ba30569 ]---
Jun 24 13:52:50 proxmox kernel: [28172.928456] RIP: 0010:poll_select_finish+0x8e/0x210
Jun 24 13:52:50 proxmox kernel: [28172.928489] Code: 00 00 04 0f 85 ae 00 00 00 49 83 3c 24 00 75 0c 49 83 7c 24 08 00 0f 84 a7 00 00 00 48 8d 7d b8 e8 e7 89 e4 ff 49 8b 54 24 08 <09> 8b 34 24 48 8d 7d c8 48 2b 55 c0 48 2b 75 b8 e8 2d 1b e4 ff 48
Jun 24 13:52:50 proxmox kernel: [28172.928601] RSP: 0018:ffffa39600547e78 EFLAGS: 00010202
Jun 24 13:52:50 proxmox kernel: [28172.928635] RAX: 0000000016dde4d1 RBX: 0000000000000000 RCX: 0000000000000018
Jun 24 13:52:50 proxmox kernel: [28172.928680] RDX: 0000000016dd2882 RSI: 0000000000006e0d RDI: 000000002ada6bb6
Jun 24 13:52:50 proxmox kernel: [28172.928725] RBP: ffffa39600547ec0 R08: 00274c0d55f78f2c R09: ffff8e333ffd2d00
Jun 24 13:52:50 proxmox kernel: [28172.928769] R10: 0000000000000000 R11: ffff8e333ffd2000 R12: ffffa39600547ee8
Jun 24 13:52:50 proxmox kernel: [28172.928814] R13: 00007ffc53ce3c50 R14: 0000000000000002 R15: 000000000000004d
Jun 24 13:52:50 proxmox kernel: [28172.928859] FS:  00007fc864d75840(0000) GS:ffff8e3337a80000(0000) knlGS:0000000000000000
Jun 24 13:52:50 proxmox kernel: [28172.928910] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 24 13:52:50 proxmox kernel: [28172.928947] CR2: ffffffff8d482434 CR3: 00000001026ae000 CR4: 00000000003426e0

I'd suggest checking the power supply and running memtest.

 

[leofabri]

Were you able to solve this? Was it a hardware problem?