Share experience with ZFS

[Tony]

Hi,

I have been running ZFS on Promox for a while and in general I am very pleased with it. I would like to share my (relatively new) experience with ZFS and hope to hear experience from other users.

I have a server SX132 at Hetzner with 10x 10TB disks, 1 zfs pool of 3 Striped 3-way Mirrored Vdevs (= 3 vdevs, each is 3-way mirror), 1 warm spare. I created ZFS vdevs on partition, not on the entire disk as usually recommended.

Then I have a dataset for VMs. All VMs are using qcow2 disks.

In general I find this setup working well for me, decent performance and I feel it's relatively safe. I would not go back to lvm.

What is your general experience with ZFS?

Regards,
Tony

 

[wolfgang]

Hi,

why do you use a 3-way mirror when you have a warm spare?
I would recommend you to use 2-way mirrors with 2 spare disks.
Also, it is not recommended to have a cow ontop a cow.
Why do you use qcow2 on top of ZFS? In the case of performance, it is very bad. one writes in the VM ends in 4 writes on the disk.

 

[UdoB]

Hello,

I have a small ZFS-only cluster in a Homelab. All vdevs are mirrors, but only with two disks, not three ;-)

For me ZFS and replication eliminates the need of shared storage. As I do not run live databases or anything else "mission critical" this is absolutely fine for my usecase. I can tolerate data loss back to the last replicaton without experiencing problems. (And of course vzdump-backups run daily/nightly, storing data on independent hardware.)

The other benefits lz4-compression and robust data integrity are highly appreciated also.

Currently I would not want to run anything else than ZFS. (I am a FreeNAS user for several years now. But I do not want to run VMs on FreeNAS as I like to have separate / independent Hypervisor and Storage services. And Proxmox is simply great!)

Thanks to the Proxmox developers - and to all other open source developers of course!

Just my 2€¢

 

[H4R0]

Im using ZOL for 5 years now. And its by far the best file system.

2 things that annoy me with proxmox.

Fragmentation on rpool is very high, i got 60% after only 3 months. This might kill performance sooner or later. I will have to monitor it deeply. But it seems to not grow that fast anymore.

And im also wondering why vm disk usage is not displayed in vm summary, not a big problem as im used to cli but its kinda weird.

 

[aaron]

And im also wondering why vm disk usage is not displayed in vm summary, not a big problem as im used to cli but its kinda weird.

PVE does not know about any partitioning and file systems on the VMs disk. Therefore, it cannot say if a block is actually referenced by the guest or if it used to be a file/block that has been deleted but the guest did not issue a trim/discard yet.

 

[H4R0]

PVE does not know about any partitioning and file systems on the VMs disk. Therefore, it cannot say if a block is actually referenced by the guest or if it used to be a file/block that has been deleted but the guest did not issue a trim/discard yet.

hmm i get it, but i do have instant trim enabled and zfs list represents disk usage inside of the vm, so it could be shown in my case without issues. But i guess many people did not setup trim/discard properly or periodic and it would show wrong values, thats a shame. IMHO it would still be great to show, as the zfs output shows the real disk usage on host which might be even more important than whats used inside of the vm.

 

[guletz]

Hi,

I have start to use zfs from curiosity, many years ago, with zfs-fuse, like this:

2012-01-23.09:11:20 zpool create myzpool /dev/md6 2012-01-23.09:12:15 zfs set compression=on myzpool 2012-01-23.09:12:43 zfs set atime=off myzpool

... and this pool works even now on the same HDD's(2 x Hitachi HUA721050KLA330, Power_On_Hours 0x0012 085 085 000 Old_age Always - 109857 ). Then, in time all my servers was running ZFS(centos mostly, manual compile). In the first 2 years zfs was a pain, and the curv learning was very bad. But after that, I can say that I like very much to have zfs. Then after I had see PMX with zfs included, I start to use it(zfs was the key factor when I chose PMX).
I had have some small problems with PMX/zfs mostly with grub(modify groub and zfs timeout have solve my problem). On all my new systems I use a separate small SSD for PMX OS only(ext4) and zfs on all HDDs. After install I create by hand zfs pool(using by-id) and setup some proprieties(discard, compression, setup a 5% reservation from the pool, and so on). Before zfs creation I test all HDDs for several days with badblocks and smartctl -t long. I create different datasets for some OS paths that I want to not "eat" my SSD OS life(/tmp, /var/log, /var/tmp, /var/lib/rrdcached) and also I use swap on zfs.
in 2 occasions/year/system I replace one HDD with a reserve one, so I can estimate the resilver time in a real case scenario, and because I want to have in the same host HDDs with different Power_On_Hours as much as is possible. In a few words any HDDs wil be stay for only several month on the same system, and then is moved on other system, in a round robin fashion.

Good luck / Bafta!

 

[Tony]

Hi,

Hi,

why do you use a 3-way mirror when you have a warm spare?
I would recommend you to use 2-way mirrors with 2 spare disks.
Also, it is not recommended to have a cow ontop a cow.
Why do you use qcow2 on top of ZFS? In the case of performance, it is very bad. one writes in the VM ends in 4 writes on the disk.

yes I have learnt that qcow2 on top of ZFS is not the best way to do that and had to convert all VMs to ZVOL. Thanks to Proxmox GUI it was not painful as it would be.

re. 3-way mirror: I wanted to have some peace in mind, and have seen some recommendation to go for 3-way mirror. It's perhaps waste of disk space this way. Anyway, another lesson for the next ZFS setup.