Best storage target for VMs and containers

Fathi

Well-Known Member
May 13, 2016
125
3
58
52
Tunis, Tunisia
Hi,
I would like to setup shared storage for my Proxmox cluster and i am wondering which is the best solution for my small cluster (no FC, no dedicated ethernet path for storage, ... just all cluster nodes and planed Trunenas core storage server with single oepnzfs pool (2 mirrored disks), on the same network): NFS, ISCSI, S3, ...
I have read several threads on Proxmox and Trunenas forums, but still can't decide how to share the nas storage (it also seams that truenas is supported via a third party community plugin which is often behind the proxmox and truenas versions ).
Can someone help me please ?
TIA.
 
I concur. I would also discuss the NAS backend. If you choose that in a way that you have ZFS-over-iSCSI, I would prefer that one.
what would be the benefits of iscsi vs nfs, knowing they are both on top of openzfs (in my case as i am planning to use truenas core) ?
I read somewhere that proxmox is issuing cli commands over an ssh connection to truenas to do various stuffs like snapshots; so, no api interaction, no real integration. It seems also that there is no interaction between the developers of these two open source projects.
so what is the benefit of ZFS-over-iSCSI over the other proposed solution (nfs) ?
 
Why not setup both and see what works for you?

It all depends on your goals, skillset and tolerance.

Home lab enthusiast? configure both and play to your hearts desire.

Want stable supported set-it-and-forget-it setup? Use NFS.

Want to learn something new, dig deep into the guts and keep on top of things - ISCSI on top of ZFS (I really think ZFS-over-iSCSI is badly misnamed).

Feature wise, I am not sure there is a big difference between Qcow/NFS/ZFS vs iSCSI/ZFS. There are probably some performance benefits as former has more layers than latter. But that all depends on NFS and iSCSI implementation, network and client tuning. You may not even notice the few microseconds difference.



Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
  • Like
Reactions: Fathi
Why not setup both and see what works for you?

It all depends on your goals, skillset and tolerance.

Home lab enthusiast? configure both and play to your hearts desire.

Want stable supported set-it-and-forget-it setup? Use NFS.

Want to learn something new, dig deep into the guts and keep on top of things - ISCSI on top of ZFS (I really think ZFS-over-iSCSI is badly misnamed).

Feature wise, I am not sure there is a big difference between Qcow/NFS/ZFS vs iSCSI/ZFS. There are probably some performance benefits as former has more layers than latter. But that all depends on NFS and iSCSI implementation, network and client tuning. You may not even notice the few microseconds difference.



Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Thank you.
 
so what is the benefit of ZFS-over-iSCSI over the other proposed solution (nfs) ?
As @bbgeek17 already mentioned, there are very feature-similar (with respect to QCOW2 on NFS), yet you have additional:
  • send/receive to replicate your ZFS
  • transparent compression
  • easier linked clones
  • possible deduplication
  • (and of course all benefits from ZFS like self-healing, silent corruption detection, etc.)
and also one drawback: just linear snapshot history with ZFS, whereas you would get tree-like snapshots in QCOW2.
 
  • Like
Reactions: Fathi

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!