VMs randomly freezing/crashing post upgrade from 8.3 to 9.1.4

[doingOK]

Hi All.
Normally i just read these and they mainly cover my needs. Having some trouble finding info about my issue. So figured i would try.

I have a 10 node cluster with about 50 VMs running. All running Ubunutu 20.04 LTS. Recently upgraded from 8.3.5(?) to 9.1.4. using the directions from Proxmox including the pve8to9 script. Script looked clean. The cluster is using local-zfs for storage on each VM, and what i noticed first was the SCSI Controller, megaRaid sas option no longer could see the VM hard disk (scsi0). Moving them to the default seemed to fix the issue. Shortly after I started getting the VMs freezing/crashing. The console would lock up and the VMs would stop responding. A reboot would resolve the issue for a short time until the next freeze < 24 hours. The VMs are running 5.15.0-138-generic kernels. Reviewing the logs nothing sticks out at all before the crash so it does not know its coming.

I could really use some help isolating the issue. I have tried rebooting the PVE nodes, making sure they are up to date, rebuilding the VMs. What logs or configs can I provide that might help figure out what i have messed up. On pve 8 this cluster and all vms ran for over a year with no issues at all so I am a bit lost because for me proxmox has always just worked. upgrading from 7to8 was a breeze back when i did it.

Thank you in advance for any help i can get.

 

[doingOK]

Example of one of the VMs that keeps freezing..

agent: 1
boot: order=net0;scsi0
cores: 30
hotplug: network,usb
ide2: none,media=cdrom
memory: 114688
meta: creation-qemu=8.0.2,ctime=1696378445
name: local-build001
net0: vmxnet3=D6:DA:18:AC:75:E6,bridge=vmbr0,firewall=1,tag=1140
numa: 0
ostype: l26
scsi0: LGStore-local:vm-144-disk-0,size=500G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=1ce109b4-f923-4f5f-9329-6d308504e869
sockets: 2
tablet: 0
vmgenid: c85f8363-e98b-462b-b828-472b91cbd40b

 

[doingOK]

the node is running :::

roxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.4 (running version: 9.1.4/5ac30304265fbd8e)
proxmox-kernel-helper: 9.0.4
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17: 6.17.4-2
proxmox-kernel-6.17.4-1-pve-signed: 6.17.4-1
proxmox-kernel-6.8: 6.8.12-17
proxmox-kernel-6.8.12-17-pve-signed: 6.8.12-17
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx11
intel-microcode: 3.20250812.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.1.3
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.4
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.0
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-3
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.1-1
proxmox-backup-file-restore: 4.1.1-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-widget-toolkit: 5.1.5
pve-cluster: 9.0.7
pve-container: 6.0.18
pve-docs: 9.1.2
pve-edk2-firmware: 4.2025.05-2
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.1.0
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-5
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.3
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1