IO Optimisation

[Znuf]

Good afternoon,

I would like to optimise IOPs on my server.
The config :
Ram 64go
4x ssd 1TO for VMs
2x HDD 4TO for in place Backup
CPU AMD Zen 16core

The VMs :
1 Windows server 2022 4 core 32go ram
3 linux with each 1 core and 1go ram

I'm on ZFS :

zpool status
  pool: Backup_HDD
 state: ONLINE
  scan: scrub repaired 0B in 01:18:35 with 0 errors on Sun Jan  9 01:42:36 2022
config:

        NAME                                      STATE     READ WRITE CKSUM
        Backup_HDD                                ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            ata-TOSHIBA_MG08ADA400E_X1G0A09CFXBG  ONLINE       0     0     0
            ata-TOSHIBA_MG08ADA400E_X1G0A09GFXBG  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:20:10 with 0 errors on Sun Jan  9 00:44:13 2022
config:

        NAME                                                   STATE     READ WRITE CKSUM
        rpool                                                  ONLINE       0     0     0
          mirror-0                                             ONLINE       0     0     0
            ata-Samsung_SSD_860_QVO_1TB_S4CZNF0MA63241V-part3  ONLINE       0     0     0
            ata-Samsung_SSD_860_QVO_1TB_S4CZNF0MA63248L-part3  ONLINE       0     0     0
          mirror-1                                             ONLINE       0     0     0
            ata-Samsung_SSD_860_QVO_1TB_S4CZNF0MA63251Z        ONLINE       0     0     0
            ata-Samsung_SSD_860_QVO_1TB_S4CZNF0MA63246M        ONLINE       0     0     0

errors: No known data errors

What can be perfectible? Or it's good like that ?

I don't need more that 1to on SSD.

Thanks a lot.

 

[Dunuin]

A striped mirror like you are using is already the fastest you can do for IOPS. Consumer SSDs, especially QLC ones like you are using, are just very slow. As soon as the cache gets full the SSDs won't be much faster than a HDD.

 

[Znuf]

Thanks a lot for these infos. Next step i'll change the SSD

 

[Dunuin]

You also didn't told us if IOPS in general or only the write IOPS are the problem. Your SSDs should be fine for read IOPS but slow on writes, especially sync writes.

Best IOPS you would get with enterprise SSD (SLC is better than MLC, MLC is better than TLC and TLC better than QLC) that got a powerloss protection in a mirror or striped mirror. And in case you got M.2 slots or free PCIe slots you might want to look for a U.2 NVMe SSD as these allow for much better IOPS than SATA.

 

[LnxBil]

Very good explanation from @Dunuin.
One addition: sync writes can be speed up by using at least one SLOG device, better two. Maybe the cheaper solution than buying 4 new SSD.

 

[Znuf]

Thanks a lot for these answeres, as you expect, it's the writes IOPS that is slow.

 

[Znuf]

I'm back here for a little question.
If I disable zfs sync : zfs set sync=disabled rpool.
my perf is perfect : 2147483648 bytes written in 0.90 seconds (2287.48 MiB/sec).

With standard zfs set sync=standard rpool :
my perf is bad : 2147483648 bytes written in 34.88 seconds (58.71 MiB/sec).

Could be a disk that is limiting my pool ?
How could I test disk by disk ?
Remove from the pool and add again ? Is it safe ?

Thanks

 

[leesteken]

I'm back here for a little question.
If I disable zfs sync : zfs set sync=disabled rpool.
my perf is perfect : 2147483648 bytes written in 0.90 seconds (2287.48 MiB/sec).

With standard zfs set sync=standard rpool :
my perf is bad : 2147483648 bytes written in 34.88 seconds (58.71 MiB/sec).

Looks like the difference is because of sync writes. Enterprise drives with PLP (Power Loss Protection) can safely cache sync writes; therefore they can do in a safe way what you do with sync=disabled in a very unsafe way. Remove all drives without PLP from your pool and you'll probably get the high speeds (with sync=standard).

Could be a disk that is limiting my pool ?
How could I test disk by disk ?
Remove from the pool and add again ? Is it safe ?

Your Samsung 860 QVO drives use QLC and will perform terrible with low IOPS and extremely low sustained write speed. They are not suitable for virtualization and/or ZFS as they perform worse than spinning harddrives (please search the forum for QLC). I suggest you remove them from the pool and use them for "mostly read-only" storage like ISO's instead.

EDIT: Don't actually do remove all non-PLP or QLC drives from your pool, as all your SSD drives are non-PLP and QLC. I don't think you can get any decent performance from your current setup, sorry.

 

[Maximiliano]

Hello, you could look into turning IO Thread on for the windows VM, it is disabled by default on Windows guests since it needs drivers to work. please see our docs [1, 2].

Unrelated to the performance, you could also enable Discard too [3, Sect. Trim/Discard].

[1] https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers
[2] https://pve.proxmox.com/wiki/Additi...rt_Windows_to_use_.28VirtIO.29_SCSI_.28KVM.29
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_trim_discard

 

[Znuf]

Thanks a lot for your support.

I have two 990 PRO PCIe 4.0 NVMe M.2 SSD. It should be good or I should buy something else ?

 

[Dunuin]

You want enterprise/datacenter grade SSDs with power-loss protection. 990 PRO are neither enterprise/datacenter grade nor do they got a power-loss protection. If you want to stick with Samsung M.2 it would be something like the PM983 and PM9A3.