Proxmox VE Crashes

[majorpaynedof]

Hello,

Please forgive me if I miss a protocol I just signed up because i'm at my wits end trying to figure out what is causing my system to just crash. The last time it was the ram sticks but I have replaced them. and I ran a memtest on the new single stick i currently have it there. Now at 3 PM today the WebGui just stopped responding, everything else on said PVE stopped working. when looking at the only logs i know I find this. Please help.


May 22 08:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
May 22 08:17:01 pve2 CRON[1801716]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 08:17:01 pve2 CRON[1801717]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 08:17:01 pve2 CRON[1801716]: pam_unix(cron:session): session closed for user root
May 22 08:42:46 pve2 systemd[1]: Starting apt-daily.service - Daily apt download activities...
May 22 08:42:47 pve2 systemd[1]: apt-daily.service: Deactivated successfully.
May 22 08:42:47 pve2 systemd[1]: Finished apt-daily.service - Daily apt download activities.
May 22 09:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57
May 22 09:17:01 pve2 CRON[1832743]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 09:17:01 pve2 CRON[1832744]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 09:17:01 pve2 CRON[1832743]: pam_unix(cron:session): session closed for user root
May 22 09:30:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
May 22 10:17:01 pve2 CRON[1863737]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 10:17:01 pve2 CRON[1863738]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 10:17:01 pve2 CRON[1863737]: pam_unix(cron:session): session closed for user root
May 22 10:30:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
May 22 11:17:01 pve2 CRON[1894689]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 11:17:01 pve2 CRON[1894690]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 11:17:01 pve2 CRON[1894689]: pam_unix(cron:session): session closed for user root
May 22 12:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
May 22 12:17:01 pve2 CRON[1925697]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 12:17:01 pve2 CRON[1925698]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 08:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
May 22 08:17:01 pve2 CRON[1801716]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 08:17:01 pve2 CRON[1801717]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 08:17:01 pve2 CRON[1801716]: pam_unix(cron:session): session closed for user root
May 22 08:42:46 pve2 systemd[1]: Starting apt-daily.service - Daily apt download activities...
May 22 08:42:47 pve2 systemd[1]: apt-daily.service: Deactivated successfully.
May 22 08:42:47 pve2 systemd[1]: Finished apt-daily.service - Daily apt download activities.
May 22 09:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 57
May 22 09:17:01 pve2 CRON[1832743]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 09:17:01 pve2 CRON[1832744]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 09:17:01 pve2 CRON[1832743]: pam_unix(cron:session): session closed for user root
May 22 09:30:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 57 to 58
May 22 10:17:01 pve2 CRON[1863737]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 10:17:01 pve2 CRON[1863738]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 10:17:01 pve2 CRON[1863737]: pam_unix(cron:session): session closed for user root
May 22 10:30:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 58 to 59
May 22 11:17:01 pve2 CRON[1894689]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 11:17:01 pve2 CRON[1894690]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 11:17:01 pve2 CRON[1894689]: pam_unix(cron:session): session closed for user root
May 22 12:00:42 pve2 smartd[887]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
May 22 12:17:01 pve2 CRON[1925697]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 12:17:01 pve2 CRON[1925698]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 12:17:01 pve2 CRON[1925697]: pam_unix(cron:session): session closed for user root
May 22 13:17:01 pve2 CRON[1956661]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 13:17:01 pve2 CRON[1956662]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 13:17:01 pve2 CRON[1956661]: pam_unix(cron:session): session closed for user root
May 22 14:17:01 pve2 CRON[1987680]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 22 14:17:01 pve2 CRON[1987681]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
May 22 14:17:01 pve2 CRON[1987680]: pam_unix(cron:session): session closed for user root
-- Boot 4eb3ed9247614fcc85e59498108997b9 --
May 22 18:32:00 pve2 kernel: Linux version 6.8.12-10-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-10 (2025-04-18T07:39Z) ()
May 22 18:32:00 pve2 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-10-pve root=/dev/mapper/pve-root ro quiet
May 22 18:32:00 pve2 kernel: KERNEL supported cpus:
May 22 18:32:00 pve2 kernel: Intel GenuineIntel
May 22 18:32:00 pve2 kernel: AMD AuthenticAMD
May 22 18:32:00 pve2 kernel: Hygon HygonGenuine
May 22 18:32:00 pve2 kernel: Centaur CentaurHauls
May 22 18:32:00 pve2 kernel: zhaoxin Shanghai
May 22 18:32:00 pve2 kernel: BIOS-provided physical RAM map:
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000009afefff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x0000000009aff000-0x0000000009ffffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x000000000a000000-0x000000000a1fffff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x000000000a200000-0x000000000a211fff] ACPI NVS
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x000000000a212000-0x00000000b00d7fff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b00d8000-0x00000000b2c35fff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b2c36000-0x00000000b2c91fff] ACPI data
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b2c92000-0x00000000b7d06fff] ACPI NVS
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b7d07000-0x00000000b957cfff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b957d000-0x00000000b95fefff] type 20
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000b95ff000-0x00000000baffcfff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000baffd000-0x00000000bbffffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000bce00000-0x00000000bfffffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x00000000fd000000-0x00000000ffffffff] reserved
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000083c2fffff] usable
May 22 18:32:00 pve2 kernel: BIOS-e820: [mem 0x000000083d340000-0x00000008601fffff] reserved
May 22 18:32:00 pve2 kernel: NX (Execute Disable) protection: active
May 22 18:32:00 pve2 kernel: APIC: Static calls initialized
May 22 18:32:00 pve2 kernel: efi: EFI v2.8 by American Megatrends
May 22 18:32:00 pve2 kernel: efi: ACPI=0xb2c91000 ACPI 2.0=0xb2c91014 TPMFinalLog=0xb7cbf000 SMBIOS=0xb93d3000 SMBIOS 3.0=0xb93d2000 MEMATTR=0xabc68218 ESRT=0xad9c0518 MOKvar=0xb938b000
May 22 18:32:00 pve2 kernel: efi: Remove mem60: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
May 22 18:32:00 pve2 kernel: e820: remove [mem 0xe0000000-0xefffffff] reserved
May 22 18:32:00 pve2 kernel: efi: Remove mem61: MMIO range=[0xfd000000-0xfedfffff] (30MB) from e820 map
May 22 18:32:00 pve2 kernel: e820: remove [mem 0xfd000000-0xfedfffff] reserved
May 22 18:32:00 pve2 kernel: efi: Not removing mem62: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
May 22 18:32:00 pve2 kernel: efi: Remove mem63: MMIO range=[0xfee01000-0xffffffff] (17MB) from e820 map

 

[l.leahu-vladucu]

Hello majorpaynedof! There's a gap of a few hours between May 22 14:17:01 (last log before the reboot) and May 22 18:32:00 (first log after reboot). Did you try to manually reboot the system? If yes, what did it show on the screen before the reboot?

Also, we would need a bit more information about the machine, including CPU, motherboard, BIOS version, storage, etc.

 

[majorpaynedof]

Hello majorpaynedof! There's a gap of a few hours between May 22 14:17:01 (last log before the reboot) and May 22 18:32:00 (first log after reboot). Did you try to manually reboot the system? If yes, what did it show on the screen before the reboot?

Also, we would need a bit more information about the machine, including CPU, motherboard, BIOS version, storage, etc.

When I turned on the monitor to the device. The screen was blank.

It is a Minisforum um690 (from 2023)
AMD Ryzen 9 6900HX with Radeon Graphics
Base Board Information
Manufacturer: Shenzhen Meigao Electronic Equipment Co.,Ltd
Product Name: F7BFC
Version: Version 1.0
Serial Number: Default string
Asset Tag: Default string
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Default string
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Samsung SSD 870 EVO 1TB S6PTNM0T519068X 931.5G running
KINGSTON OM8PGP4512Q-A0 50026B7282E6F3A4 476.9G live

Vendor: American Megatrends International, LLC.
Version: 1.11
Release Date: 12/15/2022
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 32 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 1.11

 

[l.leahu-vladucu]

Thanks for the info! Since we don't have much information about what could have went wrong, could you please:

1. Check for BIOS and other firmware updates for your Minisforum um690. The BIOS is already somewhat old. Note that you might need to reconfigure your BIOS settings after the update.
2. While I'm not so sure at this point whether this is a storage-related issue, could you please also check for firmware updates for your storage (Kingston and Samsung SSDs)? The most important is the drive you installed Proxmox VE on, of course. Note: Samsung SSDs can also be updated without Samsung Magician (which would require Windows) by downloading an ISO from this page, but I also recommend checking the Kingston SSD for firmware updates, since some recent updates for some models fixed some drives from becoming unresponsive under Linux.
3. Check smartctl -a /dev/DRIVE_NAME to see whether you see any errors (especially for the system drive).
4. You might also want to check whether installing the CPU microcode updates from Debian improves the situation.

and I ran a memtest on the new single stick i currently have it there.

How long did you let it run?

 

[majorpaynedof]

Thanks for the info! Since we don't have much information about what could have went wrong, could you please:

1. Check for BIOS and other firmware updates for your Minisforum um690. The BIOS is already somewhat old. Note that you might need to reconfigure your BIOS settings after the update.
2. While I'm not so sure at this point whether this is a storage-related issue, could you please also check for firmware updates for your storage (Kingston and Samsung SSDs)? The most important is the drive you installed Proxmox VE on, of course. Note: Samsung SSDs can also be updated without Samsung Magician (which would require Windows) by downloading an ISO from this page, but I also recommend checking the Kingston SSD for firmware updates, since some recent updates for some models fixed some drives from becoming unresponsive under Linux.
3. Check smartctl -a /dev/DRIVE_NAME to see whether you see any errors (especially for the system drive).
4. You might also want to check whether installing the CPU microcode updates from Debian improves the situation.

How long did you let it run?

First of all, Thank you this is the most information I have gotten from anyone.
1. The Bios might be a trick to update as it needs a windows system to install it. I can try to update from a VM but I doubt that will work.. maybe a boot disk....
2. I will look into
3. The Samsung no errors found. the Kingston has not had a scan done on it.
4. I ran the update and it gave me 2 options for AMD in which I picked the first...
5. RAM test.. I let it run for a full cycle. I do not recall the time. I do know I did replace 2 other sticks that were erroring out almost immediately.

 

[MarkusKo]

The Bios might be a trick to update as it needs a windows system to install it. I can try to update from a VM but I doubt that will work.. maybe a boot disk....

you can try to update the bios from within windows pe.

- create a windows 11 install usb, create a folder - example "bios" on the usb stick and copy the content of the bios update zip to the folder
- boot from the windows install usb, press "shift" "F10" to get a terminal, navigate to the bios folder (you may have to try multiple drive letters to get the correct one) and execute the "WinFlash.bat"
- after the update power off, remove usb and start your mini pc
- you should check bios settings after the update because some vendors reset all settings to default

in theory you don't need windows at all, looking at the update zip file contents there are several efi files which suggests you could format a small usb stick with fat32 and copy the content of the zip file to the stick's root. then disable secure boot in bios and try to boot from the stick directly, you may have to use some boot override within the bios setup, selecting "AfuEfix64.efi" or something similar, have not tried this myself.

i did update my um790 pro bios from windows pe successfully

 

[majorpaynedof]

The Bios might be a trick to update as it needs a windows system to install it. I can try to update from a VM but I doubt that will work.. maybe a boot disk....

you can try to update the bios from within windows pe.

- create a windows 11 install usb, create a folder - example "bios" on the usb stick and copy the content of the bios update zip to the folder
- boot from the windows install usb, press "shift" "F10" to get a terminal, navigate to the bios folder (you may have to try multiple drive letters to get the correct one) and execute the "WinFlash.bat"
- after the update power off, remove usb and start your mini pc
- you should check bios settings after the update because some vendors reset all settings to default

in theory you don't need windows at all, looking at the update zip file contents there are several efi files which suggests you could format a small usb stick with fat32 and copy the content of the zip file to the stick's root. then disable secure boot in bios and try to boot from the stick directly, you may have to use some boot override within the bios setup, selecting "AfuEfix64.efi" or something similar, have not tried this myself.

i did update my um790 pro bios from windows pe successfully

I used a Windows 11 boot disk and upgraded the bios.