100% disk utilization in Windows VMs and intermittent kernel panic every 2+ months.

[peekOfgeek]

I have been using proxmox as a home lab for a few years and absolutely love it when it works. However, I've had the following two massive issues:

• Kernel panics resulting in UI being inaccessible (sometimes you can get to the login but it just spins, other times I can't even ping)
• Windows VMs frequently show 100% disk utilization in task manager.

I can get whatever logs are needed to help me resolve these issues just point me in the right direction.

System:
Ryzen 3700
Asrock x570s riptide
64GB DDR4 3200
10gb SFP+ NIC
LSI 9217-8i
2x Samsung 850 evo zfs mirror for boot
1x 2TB Crucial SSD
5x 8TB spinners for big stuff

Windows 10 Professional VM (had similar issues with enterprise)
8gb memory
4 cores (host but I've had the issue with KVM as well)
VirtIO SCSI (yes the driver is installed)
Storage on SSDs

 

[peekOfgeek]

Bump

 

[B.Otto]

Hello,

What type of SSDs are the VMs running on? I suppose they are consumer SSDs, which are completely inadequate for both ZFS and general Hypervisor (IOPS and latency demanding) usage. For example, those Samsungs 850 Evo disks you listed are consumer SSDs.

There are a myriad of threads in this forum why you should never use consumer SSDs, but the gist is:
- they are only fast until their small SLC cache is full. Even more, SSDs with QLC tend to be as slow as normal hard disks.
- they dont have a Power-Loss-Protection (PLP). Despite the name this also means a big loss in performance, because without PLP they can't offer proper support for async writes
- they might be cheaper to buy, but are much more expensive in terms of €/TB written compared to enterprise SSDs
- it is suggested to buy some used enterprise SSDs if you don't wanna run into issues like this.

Kind regards,
Benedikt

 

[peekOfgeek]

Is this true even if I don't use ZFS?

Seems crazy proxmox would have issues with just one or two VMs running on mirrored SSDs when in the past I've had no issues running multiple VMs in esxi, hyperV, Vmware workstation etc. using just a single TLC consumer drive.

Also any recommendation on the kernel panics and what enterprise SSD do you recommend?

 

[woat]

I'm sorry but I couldn't disagree more that "consumer SSDs, which are completely inadequate for both ZFS and general Hypervisor"
We've deployed dozens of VMs running on various Hypervisors using consumer SSDs (always Mirrored or Stripe Mirror Pools) and they run fast and without issues.

The Proxmox PVE systems we've used the same drives on also work flawlessly.
All drives can die.

I'd suggest sticking to Samsung as they have been the most reliable for me.
Make sure you're using an HBA card in IT mode as well, so ZFS can see the disks directly.
You can also find these on Ebay for less than $50.

You don't need Enterprise SSDs unless you have the $$ to invest and it's a critical production system.
Even in production, if you have backups and properly mirrored storage, with drive health monitoring, you'll be in good shape.

Even the best enterprise drives die, sometimes suddenly, they are not fail proof!
@peekOfgeek
I would check your controller card, assuming you have one. If it's in RAID mode and not IT mode, that's a problem.
Also ensure the SSD are less than 80% filled as this will slow things down significantly. Writes can slow down by 50% with a nearly full SSD.
Skip the Proxmox gui and dig into your Linux processes and narrow down what the drives are doing.

If the drive is healthy and not too full, it's very likely another process is running.
You can also baseline the drive by turning off VMs and running
hdparm -Tt /dev/sdX

With a proper controller you should get 200-500mbps.