TrueNAS Core (ZFS) as Backing Storage for Proxmox: How to Set Up SSD Pool for Best Performance?

[SInisterPisces]

Hello,

I'm setting up my first TrueNAS Core box (bare metal, not VM), with 4 2.5" SATA SSDs. I want to use them to set up a networked VM storage for my Proxmox nodes so I can start learning how to use that.

I know I need to avoid a RAIDZ1 for performance reasons.

As far as how to store the VMs in the pool, I was going to set up a single pool with all 4 disks, with two mirror vdevs, and use that for (1) VM/CT backing storage that Proxmox accesses remotely; and (2) a backup destination for my production Mac. I think this is the way to go for the most space and the most performance.

Though, now I'm wondering if I shouldn't set up two separate single mirror vdev pools: one just for my production machine backup, and a second one for Proxmox network storage. Would that have performance benefits over doing a single pool with two mirror vdevs? Or does the single pool perform better in that scenario?

 

[Dunuin]

For performance you want to stripe as many mirrors as possible. So single pool with two mirrors will be double as fast as two pools with one mirror.

You probably also want to search this forum for ZFS-over-iSCSI as NFS/SMB isn't great for storing VMs.

 

[rungekutta]

Agreed, iSCSI provides the best results (vs NFS) in my own experience and I think that’s also common wisdom here and on other forums. Probably for a multitude of reasons, including that NFS invariably leads to sync writes which then requires a very fast slog to approach async performance. However also much more cumbersome, so I actually switched back to NFS anyway (where for example backups with snapshots and incremental send/receive is a breeze).

What it boils down to - Op, 4 SSDs in 2x 2-way striped mirrors will be fast, but if you end up going NFS you should also seriously consider an enterprise NVMe/U.2 drive with power loss protection for slog, otherwise this will be your bottleneck. Optane 900p or 4800x are popular alternatives and plentiful second hand. You don’t need more than 16GB or even that, so smallest size possible is fine.

What type are your other 4 drives?

 

[SInisterPisces]

Agreed, iSCSI provides the best results (vs NFS) in my own experience and I think that’s also common wisdom here and on other forums. Probably for a multitude of reasons, including that NFS invariably leads to sync writes which then requires a very fast slog to approach async performance. However also much more cumbersome, so I actually switched back to NFS anyway (where for example backups with snapshots and incremental send/receive is a breeze).

What it boils down to - Op, 4 SSDs in 2x 2-way striped mirrors will be fast, but if you end up going NFS you should also seriously consider an enterprise NVMe/U.2 drive with power loss protection for slog, otherwise this will be your bottleneck. Optane 900p or 4800x are popular alternatives and plentiful second hand. You don’t need more than 16GB or even that, so smallest size possible is fine.

What type are your other 4 drives?

Thank you both for your answers. I'll look into ZFS-over-iSCSI, but I thought that might get complicated with no de-duplication and multiple Proxmox nodes (I'm planning two nodes minimum, a high-power node and a 10-20w basic network services node)?

My 4 2.5" SATA SSDs are all SAMSUNG MZ7LH1T9 (Samsung PM 883) drives, each with a total 2.733 PBW endurance that's barely been used. I went ahead and and used the 2x2-way striped mirrors.

Unfortunately, I don't have any spare NVME slots for slog.

I have room to add 4 more drives. I was going to eventually use those slots for 3.5" 7200 RPM HDDs for mass storage (to be used for general NAS purposes in a RAIDZ1, not for Proxmox).

 

[rungekutta]

It's possible to do shared storage over iSCSI - i.e. multiple Promox nodes on top of one big iSCSI LUN on your NAS - but it requires faffing around with LVM on top of iSCSI and you lose snapshots and other features. NFS is much more straightforward, I would recommend this instead.

Do you have any spare PCIe slots? You can get PCIe to NVMe U.2 cards cheap on Amazon and they generally work fine with U.2 enterprise SSDs such as Optane 900p. Otherwise ZFS will use your PM883s for ZIL and that will be fine too, just not as fast.

Recommend avoiding de-deduplication also, unless you have very large amounts of RAM and are fairly sure it's worth the effort. What's currently in ZFS is not suitable for general purpose / extremely resource intensive. Perhaps that will change in the future, but for now best stay away.

 

[SInisterPisces]

It's possible to do shared storage over iSCSI - i.e. multiple Promox nodes on top of one big iSCSI LUN on your NAS - but it requires faffing around with LVM on top of iSCSI and you lose snapshots and other features. NFS is much more straightforward, I would recommend this instead.

Do you have any spare PCIe slots? You can get PCIe to NVMe U.2 cards cheap on Amazon and they generally work fine with U.2 enterprise SSDs such as Optane 900p. Otherwise ZFS will use your PM883s for ZIL and that will be fine too, just not as fast.

Recommend avoiding de-deduplication also, unless you have very large amounts of RAM and are fairly sure it's worth the effort. What's currently in ZFS is not suitable for general purpose / extremely resource intensive. Perhaps that will change in the future, but for now best stay away.

I'll go with NFS. Simpler is definitely preferred. I haven't figured out how to get time off work to redo my home network and storage systems. Yet.

Unfortunately, I've only got two slots in this machine: it's an SFF Dell Optiplex 5040 with an i7-6700 (8 threads) and 32 GB of DDR3-1600 RAM. One slot has my 10Gbps NIC in it, and the other has an LSI 9207-4i4e (and that's an x4 slot, so that card is only running at half speed).

I'm realizing this box is really not suitable as anything but the most barebones TrueNAS install, and that's okay while I'm learning to use TrueNAS, but eventually I'll be replacing it with something else. Possibly a TrueNAS Mini from iX Systems. The extra cost is worth having a warranty and not having to tinker. Depending on my budget at the time, I'm also looking at an HL15 from 45 Drives, but truthfully I don't really have room or budget for that right now.

For the moment, I think this box will be more than sufficient for VM storage, which is really the only thing that I absolutely need it to do.