[SOLVED] Slow ZFS performance

[hadus]

Oh. That kind of sounds like your ZFS storage doesn't have "Thin provision" enabled?

If you take a look at Datacenter → Storage → [storage pool], it should be a checkbox option in there:

View attachment 69420

Or are you thinking about LVM with thin provisioning for a different reason?

Thanks for the suggestion Thin provision is ticked off for ZFS.

I was thinking about LVM thin due to being told that ZFS mirrored would just always be slow with these cheapo SSD's I'm using (this is just a homelab thing with scavenged parts). Thought maybe LVM would be more suitable. I dont know ... maybe mirror is just not plausible for the home scenarios with normal PC hardware?

 

[justinclift]

Hmmm, I don't know that the mirror aspect of thing is really going to change much performance wise for you (good nor bad) regardless of filesystem type.

Mirroring is generally done for reliability, so if one drive craps out you're not screwed.

 

[LnxBil]

I dont know ... maybe mirror is just not plausible for the home scenarios with normal PC hardware?

Officially, there is only ZFS software supported and available in the installer, yet you can use mdadm with PVE, yet you have to do it completly manually and by using Debian if you want to have your boot drive on it. Doing it live could be possible, yet I would not try it without a proper backup and running through the steps in a VM.

 

[leesteken]

Hmmm, I don't know that the mirror aspect of thing is really going to change much performance wise for you (good nor bad) regardless of filesystem type.

Not too long ago, I added a SSD to a HDD mirror (making it a 3-way mirror) and the random reads were much better. ZFS does send more IOPS to the faster drive(s).

Mirroring is generally done for reliability, so if one drive craps out you're not screwed.

That's also why I added a third, so that my data is not instantly at risk.

 

[LnxBil]

Not too long ago, I added a SSD to a HDD mirror (making it a 3-way mirror) and the random reads were much better. ZFS does send more IOPS to the faster drive(s).

This is an exceptionally good idea ... and I never heard of it. Makes total sense.

 

[justinclift]

Cool. I've idly wondered what would happen if someone does that.

Sounds like it's an idea potentially worth exploring, to find out what the good and bad sides of doing that turn out to be.

 

[leesteken]

This is an exceptionally good idea ... and I never heard of it. Makes total sense.

Sounds like it's an idea potentially worth exploring, to find out what the good and bad sides of doing that turn out to be.

The downside is that when (not if!) the SDD fails, my PBS becomes very slow again when listing the backups or any metadata operation.
The best solution for this is using special devices but they must have at least the same redundancy as the HDD pool (since the metadata in on the special devices).
I just bough a cheap consumer SSD with TLC and am happy while it lasts. In the long run enterprise SSDs would probably be more efficient in both speed and cost, but it is not my main system and only runs once a week.

 

[igort]

What actually helped me a lot with the performance of my 8 disk raidz2 was to turn on each individual disk's cache. It was off by default so we turned it on via hdparm -W1 /dev/sdX.

You can check the disk cache via hdparm -W /dev/sdX.

It changed writing and reading 4K block (tested via fio) from around 1MB/s to around 14 MB/s.

Btw - do you think it is a normal speed for small blocks on new hardware? When i do the same fio test for 1024K blocks, i get around 700 MB/s - which is absolutely fine, i think.

Hi,

This really drastically speeds up the HDD under ZFS.
How safe is it to have write-caching enabled on an enterprise HDD in a server that is on a UPS?

Thanks

 

[LnxBil]

How safe is it to have write-caching enabled on an enterprise HDD in a server that is on a UPS?

Very unsafe, that's why the write cache is off by default.