[SOLVED] Storage Considerations: ZFS, LVM/LVMthin, File and thick vs. thin

[gnomegemini]

Hi there,

me and a friend are managing multiple gameserver, teamspeaks and low_access-websites for some friendly clans. Mostly we do it as a training for us themselves (trying new things; we are system engineers). Now we want to move on to new hardware and some questions arise:

Currently we use ZFS because it's stable and was an easy way to have an raid1 with ssd caching using ZFS. We also considered it best because of snapshot functionality. Also we're using thin provisioning because we thought it might not be bad to alloc more space to a machine to not worry in future. That was the beginning.

Now we have evolved and any service is running within a docker container. Each VM is managed by ansible using jenkins (but moving to concourse CI soon) and even the services are installed and updated using CI-Server. With rundeck the clan members can start and stop their containers and managing them by an extend so created a nice custom "game panel".

The new server has a hardware RAID5 using an own SSD-Raid1 as cache. So is ZFS still prior? What about the other storage options? Advantages over disadvantages?

Also as we only have 4-5 big virtual machines, running the docker containers we just do full backups of the volumes while the containers are stopped nightly to an NFS mount. So we don't need snapshot functionality. Also backups aren't taken as the code as infrastracture is within gitea.

Therefore do you suggest using thick provisioning to not get some surprise when a container is running away thus filling the host till crash?

Monitoring of the hosts is done via prometheus and netdata so we see when a host runs full.

As the host consumes much memory we'd also reduce ZFS RAM to ~4GiB.

What do you think guy's? Thanks in advance.

 

[ramrot]

Hardware RAID and ZFS will not work. ZFS by design is managing the disks itself, so it needs to know about the disk devices and not the device that hardware RAID will make available. Personally i would disable the hardware raid setup and configure ZFS from scratch on the new server (also RAID5 to my opinion is not ideal).

Both technologies have a lot of cons and pros, but in the end you need the best performance for your storage. A bad configured ZFS will leave you with degraded hardware. A bad hardware raid will result to a big downtime if not worst. Test both solutions, think of failure scenarios and select what make sense for you between them.

I'm always starting with giving to my vms a fixed storage size and increase/decrease depending on the case. You can also split the OS and DATA to different disks. Thin provisioning is a great feature but you only gonna get what your maximum available space is. I think that with good monitoring on the services and taking care of them, is more manageable than having vast amount of data, that needs to be cleaned up.

 

[LnxBil]

Have you enabled trimming inside of your guests (with enabled discard on the PVE disks for you VM)?

For the use of Docker in VMs, I'm running with external volumes there on NFS that are served from the PVE host itself (zfs dataset nfs export), it works great and I do not write to my VM container much. It's more space efficient than using it inside of your VM and you have the possibility to snapshot each volume individually.

 

[guletz]

Hi,

Nice setup! Congratulations.

Maybe zfs is not the fastest storage system, but for your case it could be very useful because you have compression by default. And as I only guess the storage space is limited. If actual data is compressible then could be a big advantage. You can check this before you decide to go with or without zfs.


Good luck / Bafta

 

[gnomegemini]

Many thanks for the answers,

@ramrot: Many thanks for the description ZFS vs hw raid. As my hoster is very quick in changing raid controllers, have only 1 brand and I have a backup of the config I think rebuilding is not a bad thing. Also as the git repository is on an storagebox of this provider I can recreate the whole infrastracture (core) in under 1hour and most services within 6-7h mostly automatically. Have only to get a git server running and my client. But you are right with thin provisioning. My thought was to minimize fragmentation when giving a big disk, but then it get's fragmented inside guest.

@LnxBil: Yes trimming is active. If not thin provisioning would not make sense in my opinion. NFS would only cause an additional layer of complexity and personally I find it a nightmare i.e. when userid's of host and vm are different.

@guletz: Also thanks to you. I've spoken to a Debian dev today at work and he also said it would not make sense to go with hw raid and zfs and with limited ram because then zfs is getting very slow because it needs to access the disk.

Thanks to your comments. I'll go with lvm-caching and thin provisioning. But will stick to files then.

 

[LnxBil]

@LnxBil: Yes trimming is active. If not thin provisioning would not make sense in my opinion. NFS would only cause an additional layer of complexity and personally I find it a nightmare i.e. when userid's of host and vm are different.

Yes, userid and such are a problem, but we do not have them here. The big point is snapshotable Docker volumes, it's almost like having ZFS-over-NFS.