Shared Storage - best way to connect to Proxmox?

[clicks]

Our current Proxmox cluster is connected to a single TrueNas Core Server. Most of the VMs are stored on a NFS-share, as this seemed to be an easy and usable solution at the time of installation. And actually it works quite well, but Snapshots via qcow2 aren't always nice to work with and there seem to be better options performance wise.

For our new cluster, we are going with a TrueNas M40 Server, which offers HA due to its dual controller layout. Now I have to decide how to connect Proxmox in this cluster. I have been told in various discussions that iSCSI is much more favorable in regard to stability and performance (if configured correctly). According to the Storage documentation in the wiki, ZFS over iSCSI seems to be the most versatile and flexible solution for my setup. This would actually mean ZFS on the storage side, transported via iSCSI as a block device which is used as a ZFS drive in Proxmox - an ZFS/iSCSI/ZFS sandwich so to say

Is this a viable solution, or will it cause any problems? Technically, each storage system will have to use a filesystem of any kind to store its data to provide it via iSCSI, so this might be completely normal. Did I miss any other solution?

 

[mira]

We don't have many users that use ZFS over iSCSI, but as long as you can use root on the storage side, it should be fine.
Some commands are hardcoded to use the root user on the storage side.

Most of the iSCSI users create a shared LVM on top of a big LUN, but some of the features are then not available (thin provisioning, snapshots) and in combination with multipath can lead to errors if not set up correctly or anything is changed.

If possible we rather recommend Ceph than a single storage server, but it's not always possible.

 

[Dunuin]

Some guest FS virtualized ontop of a ZFS over iSCSI ontop of ZFS sounds like alot of read/write amplification and exponential overhead.
I wouldn't be surprised if you got something like factor 100+ write amplification for sync 4K writes.

 

[clicks]

Thanks for your input. Unfortunately, Ceph is no option anymore as I couldn't find any supplier who was confident to sell such a solution and the TrueNas storage is already on its way. So what would be your choice then? Is a shared LVM above iSCSI really a better option? According to the table in the Wiki this isn't even considered a shared storage.

 

[mira]

If you can, then probably ZFS over iSCSI as it offers more features. LVM on top of iSCSI can be shared [0].

2: It is possible to use LVM on top of an iSCSI or FC-based storage. That way you get a shared LVM storage.

If you have at least 3 nodes, it's also possible to install Ceph hyper-converged [0]. Might be an option for future setups .
The benefit of such a setup is redundancy compared to a single NAS.


[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_storage_types
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pveceph

 

[bbgeek17]

A vanilla Proxmox installation will not be able to use FreeNAS for ZFS/iSCSI. Current TrueNAS/FreeNAS releases use ctld for iSCSI and native Proxmox plugin does not support it.

There is some volunteer work that has been done out of tree to enable compatibility: https://github.com/TheGrandWazoo/freenas-proxmox
I couldn't say how stable or reliable it is.

If you get the above configuration working, you don't need necessarily to create a "sandwich". The "disk/LUN" that gets exported via iSCSI to the host is seen as raw block storage. I believe a single file gets created on your TrueNAS ZFS volume that represents a LUN. You can format this resulting block storage with any filesystem you want.

Good luck.

PS some reference reading https://forum.proxmox.com/threads/ctld-support-for-zfs-over-iscsi.29811/

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[clicks]

Thanks a lot for the link, didn't know about this project. Fascinating idea - although I'm not sure if I want/should use it in production. Otherwise, the ZFS over iSCSI implementation seems to be born out of a similar community effort.

Damn, I wish I had seen your blockbridge signature earlier, would have loved to discuss your product as I was looking for a storage a few month ago.