This is a long shot, but I figured I'd put it out there to see if anyone else is having the same issues.
I run a 6 node cluster, it's been running for 6 months or so with no issues on Proxmox VE 8. All nodes are using AMD EPYC 7713, 512GB / 1TB RAM, and Gigabyte MZ32-AR0 boards. About 2 weeks ago, I ran apt upgrade on all of the nodes and rebooted them all to load the updated kernel. I frequently update the nodes via apt, but this was the first time in a while that I had rebooted them, so a new kernel (6.5.13-3) was loaded.
Since upgrading these, I've had the same issue on 5 out of my 6 nodes. The issue is very strange, and random, but essentially the node just completely 'hangs' until the ipmi watchdog reboots it because it's timing out (or I force reboot it). dmesg shows no logs anytime before the reboot, and I can't find any other logs anywhere else. It doesn't seem to coincide with load at all, it's happened on nodes with 10% CPU / RAM used, and ones that are almost full.
Anyway, I'm sure this is some deeper hardware bug, but was just wondering if anyone else had issues with this specific kernel version. This is the only significant change that I would have guessed would cause something like this (or maybe the pve-firmware update also??). Before loading this kernel, I hadn't experienced anything like this. I updated them to 6.5.13-5 today, but debating updating to the 6.8 branch.
I run a 6 node cluster, it's been running for 6 months or so with no issues on Proxmox VE 8. All nodes are using AMD EPYC 7713, 512GB / 1TB RAM, and Gigabyte MZ32-AR0 boards. About 2 weeks ago, I ran apt upgrade on all of the nodes and rebooted them all to load the updated kernel. I frequently update the nodes via apt, but this was the first time in a while that I had rebooted them, so a new kernel (6.5.13-3) was loaded.
Since upgrading these, I've had the same issue on 5 out of my 6 nodes. The issue is very strange, and random, but essentially the node just completely 'hangs' until the ipmi watchdog reboots it because it's timing out (or I force reboot it). dmesg shows no logs anytime before the reboot, and I can't find any other logs anywhere else. It doesn't seem to coincide with load at all, it's happened on nodes with 10% CPU / RAM used, and ones that are almost full.
Anyway, I'm sure this is some deeper hardware bug, but was just wondering if anyone else had issues with this specific kernel version. This is the only significant change that I would have guessed would cause something like this (or maybe the pve-firmware update also??). Before loading this kernel, I hadn't experienced anything like this. I updated them to 6.5.13-5 today, but debating updating to the 6.8 branch.
