I/O delay with SSDs

Ser Res

New Member
Oct 3, 2022
3
1
1
Yesterday such a story happened. There are I/O latencies. Didn't install anything. Drives Samsung SSD EVO Plus 250 Gb, SSD 870 QVO 4Tb, RAM 32 Gb, 4*Intel Core i5-6500. S.M.A.R.T SSDs are good. What is the reason and how to fix it. This may have happened after a Proxmox update.
 

Attachments

  • prox1.png
    prox1.png
    52.3 KB · Views: 48
QLC SSDs as your Samsung QVO are slow as hell (even slower than a HDD for sequential writes) when writing a lot of data, so IO delay might ramp up really high as soon as the fast the RAM cache and SLC cache of the SSD get full. You really don't want to use any QLC SSDs with server workloads. Don't cheap out on SSDs when building a server.

So I would first check if your workload was writing a lot after 21 o'clock. Like a WinVM doing an upgrade, some torrenting or similar that writes lot.
 
Last edited:
  • Like
Reactions: spirit
QLC SSDs as your Samsung QVO are slow as hell (even slower than a HDD for sequential writes) when writing a lot of data, so IO delay might ramp up really high as soon as the fast the RAM cache and SLC cache of the SSD get full. You really don't want to use any QLC SSDs with server workloads. Don't cheap out on SSDs when building a server.

So I would first check if your workload was writing a lot after 21 o'clock. Like a WinVM doing an upgrade, some torrenting or similar that writes lot.
The graph shows that until October 1, everything was fine. I repeat: I did not install or remove anything. it most likely happened at night.
 
  • Like
Reactions: mrdoodle
The graph shows that until October 1, everything was fine. I repeat: I did not install or remove anything. it most likely happened at night.
Maybe all the flash memory is in use now (because the drive does not know that some blocks are no longer in use) and it has to clear each block (very slowly) while you are writing more data.
Are those drives regularly trimmed? If not, then it accumulates old deleted data until you eventually get write performances issues.
What filesystem did you use for Proxmox? What storage are you using for your VMs? Is discard enabled for all VMs? Is discard/tim enabled in the operating systems inside the VM?
 
Last edited:
Many containers and virtual machines have been launched. Everything was fine. now running one virtual machine and such problems.
 
did you get to the bottom of this? i have the same issue, it was running fine, then over night we are seeing high latency across our SSD pool. 8 drives with high latency but its not consistent.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!