Best storage target for VMs and containers

[Fathi]

Hi,
I would like to setup shared storage for my Proxmox cluster and i am wondering which is the best solution for my small cluster (no FC, no dedicated ethernet path for storage, ... just all cluster nodes and planed Trunenas core storage server with single oepnzfs pool (2 mirrored disks), on the same network): NFS, ISCSI, S3, ...
I have read several threads on Proxmox and Trunenas forums, but still can't decide how to share the nas storage (it also seams that truenas is supported via a third party community plugin which is often behind the proxmox and truenas versions ).
Can someone help me please ?
TIA.

 

[bbgeek17]

Use NFS

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[JasonJoel]

I'm not sure there is a real/conclusive "right answer" for this. But I would use NFS.

 

[LnxBil]

I'm not sure there is a real/conclusive "right answer" for this.

I concur. I would also discuss the NAS backend. If you choose that in a way that you have ZFS-over-iSCSI, I would prefer that one.

 

[Fathi]

I concur. I would also discuss the NAS backend. If you choose that in a way that you have ZFS-over-iSCSI, I would prefer that one.

what would be the benefits of iscsi vs nfs, knowing they are both on top of openzfs (in my case as i am planning to use truenas core) ?
I read somewhere that proxmox is issuing cli commands over an ssh connection to truenas to do various stuffs like snapshots; so, no api interaction, no real integration. It seems also that there is no interaction between the developers of these two open source projects.
so what is the benefit of ZFS-over-iSCSI over the other proposed solution (nfs) ?

 

[bbgeek17]

Why not setup both and see what works for you?

It all depends on your goals, skillset and tolerance.

Home lab enthusiast? configure both and play to your hearts desire.

Want stable supported set-it-and-forget-it setup? Use NFS.

Want to learn something new, dig deep into the guts and keep on top of things - ISCSI on top of ZFS (I really think ZFS-over-iSCSI is badly misnamed).

Feature wise, I am not sure there is a big difference between Qcow/NFS/ZFS vs iSCSI/ZFS. There are probably some performance benefits as former has more layers than latter. But that all depends on NFS and iSCSI implementation, network and client tuning. You may not even notice the few microseconds difference.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Fathi]

Why not setup both and see what works for you?

It all depends on your goals, skillset and tolerance.

Home lab enthusiast? configure both and play to your hearts desire.

Want stable supported set-it-and-forget-it setup? Use NFS.

Want to learn something new, dig deep into the guts and keep on top of things - ISCSI on top of ZFS (I really think ZFS-over-iSCSI is badly misnamed).

Feature wise, I am not sure there is a big difference between Qcow/NFS/ZFS vs iSCSI/ZFS. There are probably some performance benefits as former has more layers than latter. But that all depends on NFS and iSCSI implementation, network and client tuning. You may not even notice the few microseconds difference.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thank you.

 

[LnxBil]

so what is the benefit of ZFS-over-iSCSI over the other proposed solution (nfs) ?

As @bbgeek17 already mentioned, there are very feature-similar (with respect to QCOW2 on NFS), yet you have additional:

• send/receive to replicate your ZFS
• transparent compression
• easier linked clones
• possible deduplication
• (and of course all benefits from ZFS like self-healing, silent corruption detection, etc.)

and also one drawback: just linear snapshot history with ZFS, whereas you would get tree-like snapshots in QCOW2.