[SOLVED] Very bad fsync and performance on SSD root pool

[pbo10]

Hi,

I've been using Proxmox on and off for maybe a year now but only recently started to get a proper system set up and add more VMs so I haven't noticed any issues until now, because of that I'm not sure if they have always existed or whether they came in from a recent change/update. This system was originally installed I think as Proxmox 6.0.0 and has always been updated to the latest as new releases came out. Now that I have around 5 VMs running I've realised as soon as one of them is writing to the disk everything else starts to freeze up.

I did some reading on here and ran the pveperf command which is giving me the following for rpool which consists of 2 SSD drives in a ZFS mirror:

root@r730xd:~# pveperf
CPU BOGOMIPS: 115205.52
REGEX/SECOND: 2197344
HD SIZE: 281.37 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND: 15.32
DNS EXT: 17.69 ms
DNS INT: 180.53 ms ()

Note that the above test was run without a single VM running. If I fire up 5 VMs and run pveperf it doesn't even finish running, it gets to the point where it lists HD Size and then just hangs, I've tried leaving it 10mins plus but it never finishes.

For comparison I have an 8 drive ZFS z3 array in the same system consisting of 8TB WD Red 5200RPM drives:

root@r730xd:~# pveperf /HDD
CPU BOGOMIPS: 115218.36
REGEX/SECOND: 2131442
HD SIZE: 17756.89 GB (HDD)
FSYNCS/SECOND: 113.01
DNS EXT: 17.69 ms
DNS INT: 659.25 ms ()

Surely 5200rpm HDDs should not have a better result than mirrored SSD drives? Something must be wrong here right?

To give some information about the system, it's a Dell R730XD with 128GB memory. The SSDs are Advantech and according to the spec sheet have the following stats which I don't think are terrible enough to warrant what I'm seeing:

Sequential:
550 / 530
Burst IOPS@4K: 100K / 100K
Sustain IOPS@4K: 10K / 10K

Has anyone got any ideas? Do I just need to replace the SSDs with faster drives? Is it something in the configuration I need to look at?

I'm not really sure where to start, so any help would be appreciated.

 

[LnxBil]

root@r730xd:~# pveperf /HDD
[...]
FSYNCS/SECOND: 113.01
[...]

That is the expected IOPS you get with one vdev raidz, so this is fine.

I've never heard of the SSD manufacturer before and I if I search on google this thread comes up, besides nothing else for that matter. Is there another description of the SSDs? A good buyers guide is always Sebastiens blog (enterprise SSDs).

 

[pbo10]

I think I can mark this as solved now. I realised I had 2x Samsung 860 SSD drives not doing anything so swapped them out and now get 800 FSYNCS/SECOND with no VMs running on the same system. Even with my VMs running it sits around 300 and everything runs much better so I think it was just the original SSDs I was using for the rpool. I hadn't realised quite how significant the difference between SSDs could be.

Thanks for the input though, its interesting reading that link seeing how big the difference between consumer and enterprise SSDs actually is. I think I may be due an upgrade in the future!

 

[sigo]

I think I can mark this as solved now. I realised I had 2x Samsung 860 SSD drives not doing anything so swapped them out and now get 800 FSYNCS/SECOND with no VMs running on the same system. Even with my VMs running it sits around 300 and everything runs much better so I think it was just the original SSDs I was using for the rpool. I hadn't realised quite how significant the difference between SSDs could be.

Thanks for the input though, its interesting reading that link seeing how big the difference between consumer and enterprise SSDs actually is. I think I may be due an upgrade in the future!

For my PM983 SSD NVMe (zfs mirror) I have approx. 15000 fync/sec.