Ideas for large ZFS array

[totalimpact]

I am up in the air on which platform to go with - Dell or Quanta. I have done some limited ZFS installs to date so I know my way around it for the most part, but never anything big. I need at least 16TB of fast space, with room to grow later.

So typically my go-to is a Dell R730xd 26x 2.5 bays.... but this limits me to pricey/small 2.5 drives, I know all the variants of this machine pretty well, like the 12 bay etc.

Point being, I can get a Quanta Stratos S210-X22RQ
12x 3.5 + 2x 2.5 192GB RAM for less $$ than a Dell with 32gb RAM.

workload:
I have a Windows server, 2 Win7 vms, 2 linux VMs, nothing is very write intensive, but users want to store a boatload of photos on it. Currently this workload is running on a Dell R710 all SAS spinners, 32GB RAM.

Plan on the Quanta:
2x Samsung M.2 sticks on a PCIe card for PVE OS only, ZFS MIrror.
2x Intel Optane 280gb for ZIL/SLOG
6x HGST He8 8TB ZFS RAID10 (is 6x enough spindles?)

That leaves me room to at least double my storage in the future.

The old R710 will get re-purposed with 4-6 more 8tb drives +RAM, to do PVE-Zsync in another building,

I have been over these for reference:
https://forum.proxmox.com/threads/zfs-vs-other-fs-system-using-nvme-ssd-as-cache.33845/

https://forum.proxmox.com/threads/optimum-zfs-configuration-with-8-slots.32052/

https://www.servethehome.com/buyers...as-servers/top-picks-freenas-zil-slog-drives/

 

[pep]

Hi, we have just finished our setup.
look at "ssg-5029p-e1ctr12l-2u-based-nas-review-request.58561" on freenas forum.
(take a look at the link provided by the freenas user for another setup).

It is based on a supermicro system with 6*10TB sas drives raid-6, 2*1TB ssd samsung evo and 128GB ram ECC.
We run freenas on the pve 5.1 with pci-passthrough, 3 windows (2016+2012R2*2) + 1 big linux server (multiple dockers).
The performance are just enormous.
We have dedicated 64GB ram for the freenas that serve the files and reach 110MB/s read throughput on 1Gb link (!).
I think the secret there si having enougth RAM to cache for read, the other point (number of disk etc... are details I think.)

The freenas use:
6*10TB He10 drives SAS (but not sure it is that important for performances), used in pci passthrough mode (so dedicated to it)
1 nvme ssd 256GB samsung for read cache, used in pci passthrough mode. (I think this one is not necessary due to the 64GB of ram dedicated to it)
we do not use ZIL device, because of the need of costly ssd enterprise device, and performance are just enougth

The other vm run on 2*1TB 850 evo drive mirror mode, no problem.
We have used 2 small 32GB ssd for pve in zfs mirror mode because we had some, and they work perfectly, but your solution seems perfect.

The most important point is using ECC ram.

From my point of view, if you have lot of ram, ZIL/Log is not necessary (at least from what I'm seeing).
At worse you can add it later

So your setup seems very correct. We have also choosen 3.5" because the price/capacity is way much lower, like you said.

Hope it helps.

 

[totalimpact]

Thanks so much for this, it is exactly the encouragement I was looking for.

 

[LnxBil]

I need at least 16TB of fast space, with room to grow later.

This implies for me 15k SAS or all SSD (which is cheaper nowadays), yet I was mistaken If I read that 2,5'' are costly ... no (enterprise grade) server I saw in the last 5 years had 3,5'' disks in it. If speed is the key, you need a lot of spindles or SSD.

...2*1TB ssd Samsung evo and 128GB ram ECC.

Please test the super cheap Samsung evo. They're rubbish for enterprise use, wearout is just enormous in a few month, and I/O is extremely slow in comparison to good SSDs. I also bought twice :-/

Please consider buying enterprise-grade hardware like the ones on this list:

http://www.sebastien-han.fr/blog/20...-if-your-ssd-is-suitable-as-a-journal-device/

 

[pep]

I just took a look today to ARC cache (read) and the evo SSD for read cache seems not useful at all. So far, thanks to the 64GB or RAM allocated to zfs it hasn't been used.
One other point when a ZIL device can be useful if your user transfer big file on one time transfer. I've seen my write throughput going low when doing large transfer (hundred GB)... yeah the spinning disk will have to keep up at one moment. An 256GB optane disk can be quite useful in this case.
I think I'll add this to my system later.

Ha and, the hardest thing I add to workaround was to be able to boot on ZFS mirror. I had to add a sleeptime of 10s on linux command line, to let enough time to system to "see" the ZFS volumes.

Hope it will helps!

 

[LnxBil]

One other point when a ZIL device can be useful if your user transfer big file on one time transfer. I've seen my write throughput going low when doing large transfer (hundred GB)... yeah the spinning disk will have to keep up at one moment. An 256GB optane disk can be quite useful in this case.
I think I'll add this to my system later.

A ZIL does not solve all performance and write issues. Please read up what a ZIL does, how it does it and how to size it. 256 GB is way too much

Often people complain about the slowness of harddisks, yet copying large files is what harddisks are best for. You can reach GB/sec (not bit, byte) without much trouble in a large array of disks (e.g. >=8).

Ha and, the hardest thing I add to workaround was to be able to boot on ZFS mirror. I had to add a sleeptime of 10s on linux command line, to let enough time to system to "see" the ZFS volumes.

This was with the default PVE iso image? One thing that can happen is when you have a lot of DAS shelves on your HBA that needs to be scanned. Easier could it be - we do this - to have the system on its internal drives (mostly two SAS 10k drives, RAID1) and the pool completely externally only attached to the HBA. In case of a total server failure, we just swap out the 1 HE server, replug the shelves and are good to go.