LXC very slow, PVE high io delay

[sprickw]

I've got two LXC on the same PVE, one running a script accessing a database server running on the other LXC. Did that several times, but possibly using another VE-Server months ago.

Now progress of the script is VERY slow. Looking into the LXCs everthing seems fine, CPU, RAM, network, disk io - all low.

But summery of PVE shows very hight IO delay. Please see attached file.

PVE was installed on Samsung SSD 870 QVO with 1 TB. I did read something about problems with SSD here and at other places - but always thought that SSD should improve performance.

One of the LXC does also use that SSD, the other does not.

Any hints? I'm not to deep into system internals and hardware, so please KISS.

Greetings Wolfgang

 

[LnxBil]

But summery of PVE shows very hight IO delay.

That is a lot. Maybe try to monitor which process is causing those high values. Please install iotop and run it on your PVE host. It'll show what process is causing a lot of I/O. Maybe the swapping is also a problem? This seem high despite having a lot of RAM free.

 

[leesteken]

PVE was installed on Samsung SSD 870 QVO with 1 TB. I did read something about problems with SSD here and at other places - but always thought that SSD should improve performance.

SSDs with QLC flash memory (like yours) can become much slower that spinning HDD under heavy/constant writes. Lots of issues when used with ZFS (for example) can be found on this forum.

 

[sprickw]

SSDs with QLC flash memory (like yours) can become much slower that spinning HDD under heavy/constant writes. Lots of issues when used with ZFS (for example) can be found on this forum.

I don't use ZFS, only LVM and EXT4. And I consider writes < 1 MB/sec not being heavy. :-(

 

[sprickw]

SSDs with QLC flash memory (like yours) can become much slower that spinning HDD under heavy/constant writes. Lots of issues when used with ZFS (for example) can be found on this forum.

That is a lot. Maybe try to monitor which process is causing those high values. Please install iotop and run it on your PVE host. It'll show what process is causing a lot of I/O. Maybe the swapping is also a problem? This seem high despite having a lot of RAM free.

Installed iotop. See attached file with screenshot. It dows show my database process and my script, but I consider overall load to be quite low.

 

[leesteken]

I don't use ZFS, only LVM and EXT4. And I consider writes < 1 MB/sec not being heavy. :-(

With ZFS you would get write errors and QLC drives would be next to impossible to use.
With ext4 and LVM you get very slow writes (down to KB/s) when the drive does not go idle often enough. They can be used but are terrible for virtualization with multiple systems (VM/CT) worth of IO operations. Best used for ISO storage and other stuff that does not change often. Many threads of complaints about QLC drives on this forum (not only ZFS).

EDIT: I fear that you'll never be happy with that drive and don't know how to fix its slowness. TLC drives can be good enough, enterprise SSDs with PLP are the best.

 

[LnxBil]

And I consider writes < 1 MB/sec not being heavy. :-(

Depending on the circumstances, this can be VERY HEAVY: Beeing in a one harddisk setup with a random write pattern will yield for a 15k SAS drive (so the FASTEST enterprise harddisk there is) roundabout 200 IOPS at the default blocksize of 512 bytes in sum 100 KB/s and have the maximum IO load you can have with a fully queue and 100% IO delay. Comparing that to the lowest end SSD possible (yours) it could already be at the maximum possibiliy as @leesteken already pointed out.

See attached file with screenshot.

thank you, now please restrict it to only active processes that're doing IO with -oPak and let it run for some time to get a few more entries.

 

[sprickw]

Additional information as requested (iotop -oPak) see attached file.

So "SSDs with QLC flash memory" is a problem. Not easy to fix, needs re-installation of Proxmox VE I presume. No problem.

I could move the LXCs incorporated to some other server using M.2 storage. Same problem there?