Proxmox VE 8.2.4 random guest freezes

[StimzRx]

Just made an account to report that I was having a seemingly very similar issue myself. Fresh Proxmox install, fresh VM of PopOs(Latest) with GPU passthrough of a NVIDIA RTX 2070 Ti. Default storage settings. Between 5min-16hr or so after startup the guest either reboots or completely freezes and doesn't respond to the host web UI shutdown or restart requests. The "STOP" button in the web panel does seem to work to shut it down, though. Disabling iothread does not seem to have any effect.

 

[fiona]

Hi,

Just made an account to report that I was having a seemingly very similar issue myself. Fresh Proxmox install, fresh VM of PopOs(Latest) with GPU passthrough of a NVIDIA RTX 2070 Ti. Default storage settings. Between 5min-16hr or so after startup the guest either reboots or completely freezes and doesn't respond to the host web UI shutdown or restart requests. The "STOP" button in the web panel does seem to work to shut it down, though. Disabling iothread does not seem to have any effect.

GPU passthrough can always be a bit finicky. Do you experience the same freeezes without that? Do you see any errors in the host or guest's system logs/journal?

 

[fiona]

https://forum.proxmox.com/threads/what-is-kvm-desc-next-is-3-indicative-of.128739/page-3#post-688864

When we run IO heavy VMs on thread enabled disks the VM freezes.
Thats why we run all our database servers without iothread since a few month.
This one change to "iohtread=0" stabelized all our database VMs. (Widwos and Linux)

So i thought he could try it.

I'm not sure it's the same issue as the linked one, because that seems to be specific to recent versions of VirtIO Windows guest drivers.

It's true that there were issues with iothread in the past that could deadlock the whole QEMU process, but at the latest since QEMU 8.1/9.0 the situation should be much better. Should you want to debug the issue further (with a test VM enabling iothread), please also share the information that was requested earlier after a freeze happens: https://forum.proxmox.com/threads/proxmox-ve-8-2-4-random-guest-freezes.151294/post-685326

 

[BenediktS]

@fiona: I will build a test scenario next week with a linux mysql db, to see if my claim is still right.

@WoodBench: Had you have another freeze ?

 

[truonganhtuan]

Have you updated the microcode? I used to have the same problem as the OP's but it was fixed after I updated the microcode.

 

[WoodBench]

@fiona: I will build a test scenario next week with a linux mysql db, to see if my claim is still right.

@WoodBench: Had you have another freeze ?

I had a freeze last night. Stopped the VM, disabled IO Thread and started the VM again.

 

[WoodBench]

Have you updated the microcode? I had the same problem as the OP's but it is fixed after upgrading the microcode.

No, apparently you need contrib, non-free and non-free-firmware enabled in /etc/apt/sources.list, and since the were not present in my system by default, I had to add them to update/install intel-microcode.
I've done that now, thank you for the suggestion.

dmesg | grep microcode
[    0.314258] Register File Data Sampling: Vulnerable: No microcode
[    0.314259] SRBDS: Vulnerable: No microcode
[    1.002581] microcode: Current revision: 0x24000023

# Added necessary sources in sources.list and updated intel-microcode

dmesg | grep microcode
[    0.315114] SRBDS: Vulnerable: No microcode
[    1.010564] microcode: Current revision: 0x24000026
[    1.010566] microcode: Updated early from: 0x24000023

 

[truonganhtuan]

No, apparently you need contrib, non-free and non-free-firmware enabled in /etc/apt/sources.list, and since the were not present in my system by default, I had to add them to update/install intel-microcode.
I've done that now, thank you for the suggestion.

dmesg | grep microcode
[    0.314258] Register File Data Sampling: Vulnerable: No microcode
[    0.314259] SRBDS: Vulnerable: No microcode
[    1.002581] microcode: Current revision: 0x24000023

# Added necessary sources in sources.list and updated intel-microcode

dmesg | grep microcode
[    0.315114] SRBDS: Vulnerable: No microcode
[    1.010564] microcode: Current revision: 0x24000026
[    1.010566] microcode: Updated early from: 0x24000023

I'm keeping my finger crossed that everything will be fine ^^

 

[WoodBench]

@truonganhtuan So far so good. I'll give it a couple of days and if its still stable then try do enable IO Threads again. If it's still stable, then you solved the problem (and saved my sanity)!
Not being able to receive the newest intel-microcode using Proxmox VE default apt sources could pose a problem for other users as well. The information can be found on the Proxmox Wiki. According to this post the Proxmox team might upload intel-microcode to their own repositories in the future.

 

[RoCE-geek]

Hi @WoodBench, I'm not sure if this can help, but you can try downgrade your virtio scsi driver as I propose here: https://forum.proxmox.com/threads/r...system-unresponsive.139160/post-691337

 

[WoodBench]

Hi @WoodBench, I'm not sure if this can help, but you can try downgrade your virtio scsi driver as I propose here: https://forum.proxmox.com/threads/r...system-unresponsive.139160/post-691337

Cheers, I'll try that if the intel-microcode doesn't come through, I have a good feeling about it though. I enabled IO Threads a couple of days ago with no freezes so far. Unfortunately I did have to reboot this morning because a non-related self inflicted now solved disk space issue. If my server make through this weekend I'm inclined to declare the problem solved and outdated intel-microcode the culprit.

 

[fiona]

Cheers, I'll try that if the intel-microcode doesn't come through, I have a good feeling about it though. I enabled IO Threads a couple of days ago with no freezes so far. Unfortunately I did have to reboot this morning because a non-related self inflicted now solved disk space issue. If my server make through this weekend I'm inclined to declare the problem solved and outdated intel-microcode the culprit.

In the first post you mention running Ubuntu. Those drivers are specific for Windows.

 

[WoodBench]

In the first post you mention running Ubuntu. Those drivers are specific for Windows.

@fiona Tbh I haven't really looked into @RoCE-geek suggestion as my guest system runs smoother than butter on a hot pancake after offering the Proxmox spirits updated microcode. Thank you for mentioning it, though.

 

[WoodBench]

So the guest has been rock solid since I updated intel-microcode. Thank you @truonganhtuan for the tip and @fiona for help with the debugging.