Server Proxmox freezes randomly

[jp22]

Good afternoon, I'm having issues with a Proxmox server that randomly freezes and requires manual reboot. Any ideas? Thanks


Proxmox Version 8.0.3
PowerSupply = new
Clean install with new disk ssd wd green
Only one VM winserver2019
MotherBoard: GA-B250M-D3H

*Dmesg Attached

Before hung journctl says

16 19:17:01 prox1 CRON[945589]: pam_unix(cron:session): session closed for user root
Aug 16 19:25:00 prox1 smartd[1580]: Device: /dev/sda [SAT], is back in ACTIVE or IDLE mode, resuming checks (1 check skipped)
Aug 16 19:25:00 prox1 smartd[1580]: Device: /dev/sdc [SAT], is back in ACTIVE or IDLE mode, resuming checks (4 checks skipped)
Aug 16 19:55:05 prox1 smartd[1580]: Device: /dev/sda [SAT], is in STANDBY mode, suspending checks
Aug 16 19:55:11 prox1 smartd[1580]: Device: /dev/sdc [SAT], is in STANDBY mode, suspending checks

"At 20:00 PM, it stopped working, and a physical server reboot was required. no ssh, no gui, nothing only physical reboot"


Config vm:
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
agent: 1
boot: order=scsi0;net0
cores: 6
cpu: x86-64-v2-AES
machine: pc-i440fx-8.0
memory: 14000
meta: creation-qemu=8.0.2,ctime=1691876077
name: WINSRV2019
net0: virtio=5E:6A:2F:14:71:2F,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
scsi0: VolumeZFS:vm-100-disk-0,cache=writeback,discard=on,iothread=1,size=150G
scsi1: VolumeZFS:vm-100-disk-1,cache=writeback,discard=on,iothread=1,size=150G
scsihw: virtio-scsi-single
smbios1: uuid=eeb479e4-7bef-4dcc-ab5a-e87f7932692a
sockets: 1
usb0: host=4971:1020,usb3=1
vmgenid: 0a4dc3a9-4654-433e-bfd4-18c2e2f87c7f

 

[Lukas Wagner]

Hmmm, I don't see anything particularly useful in the logs. Kernel seems up-to-date. Have you checked for any available BIOS upgrades?
Could you also send me the output of:
pveversion -v

 

[AviCA]

I do have the same problem so it seems.

Last week I completed in place upgrade from 7 to 8 using the instruction here.

Attached syslog and output of:
last -x

and output of
pveversion -v

After upgrade I had issue with system time which I managed to correct.
Then I had issue with Martian source in Frigate CT which vanished after few days.
Since freezes continued and after I red this post, I checked memory with Memtest but found nothing.
Then I suspected OpnSense which run on one of my VM and disable it and start using standalone DHCP, but again, the freezes continued.
Finally I removed the Ethernet card which I installed just before upgrade, guessing it might cause some PCI compatibility issues. Again the freezes continued.

I am posting to get some more ideas what to try next.
I am thinking about downgrading back to Proxmox 7 but can't find instructions how to do that.
Also thinking about clean install but not sure how to do that since I have TrueNAS in one of the VM with it's own ZFS volume (Pass Through Disks).

After reading this post I updated the BIOS which was outdated. Will report if the freezes stop.

 

[_gabriel]

what cpu ?
try disabling ksm.

 

[AviCA]

Core i9 11900T

Not sure what is ksm.

 

[_gabriel]

try Disabling KSM then Reboot
https://pve.proxmox.com/wiki/Kernel_Samepage_Merging_(KSM)

 

[AviCA]

@_gabriel. Thank you.
I will wait to see if after the BIOS update I get the freezes.

 

[scyto]

any Idea why the drives are going into standby, I see nothing like that in my logs?

 

[AviCA]

3 days after BIOS upgrade the system freeze again. Disabled KSM right now. Will report.

 

[AviCA]

Had today two events of freeze after disabling KSM. Any more ideas?
I am running now the PVE with all VM and CT down to test if freeze repeats.

 

[cnupy]

did you find a solution?

 

[AviCA]

It had to do with Frigate and MQTT on the same PVE. I found a thread that speak about that. So I removed Frigate to another node and that did it.