Need clarification on ZFS/iSCSI shared storage

[Marco2G]

Hi everyone

I've tried to google this but my Google Fu has been moved into legacy land by Google lately. I hat a talk with ChatGPT about this but that was not completely resolving all issues.

I have a ESXi setup running currently. My vCenter crashed and even older Restores no longer work. I'm also tired of using Veeam for backups with temporary licenses. With the whole Broadcom fiasco, I thought I'd try to move my homelab to Proxmox... future proofing my knowledge and such.

So I have two Proxmox hosts now and I successfully mounted an iSCSI LUN from TrueNAS (a zVol on a three stick NVMe ZFS). Upon reading about this topic on the net, I was left with the impression that there is a thing like ZFS over iSCSI. I have the impression that this might give Proxmox the ability to leverage TrueNAS' ZFS for snapshots and backups?
Or am I misunderstanding and the idea would be to just put the iSCSI LUN into a local ZFS? As in TrueNAS ZFS > zVol > iSCSI Target > iSCSI Lun Proxmox > ZFS datastore?

Either way, what would be pros and cons? What is thesleakest, most efficient and perhaps nerdy way to do this? Could I somehow mount the TrueNAS zVol directly for every VM and have ZFS snapshoting used for individual VM backups?

If this has been answered a hundred times I apologize and would appreciate a link for further reading. The Proxmox Wiki so far hasn't delivered what I am looking for (again, I'm no good at search functions anymore...)

Edit: Upon further reading I think I see where this is going. I'm currently trying to move a VM disk from local LVM to this iSCSI and failing so far. The disk is RAW and as I understand it, that necessitates a datastore that has snapshot capability. Which LVM does AFAIK.

So is my assumption correct that I either need to use QCOW2 VM disk format if I want to use bare iSCSI? Or alternatively mount the iSCSI as an LVM or ZFS to regain snapshot capability and there is no scenario in which proxmox leverages snapshots on the TrueNAS sside at all?

Thanks

 

[bbgeek17]

I have the impression that this might give Proxmox the ability to leverage TrueNAS' ZFS for snapshots and backups?

It can, in theory.

iSCSI LUN into a local ZFS? As in TrueNAS ZFS > zVol > iSCSI Target > iSCSI Lun Proxmox

This is exactly how it works, note the omission of the last part of your diagram.

The management (create, delete, snapshot, etc) is offloaded to a special storage plugin that implements "ZFS/iSCSI". Each resulting iSCSI LUN is passed through as raw disk to a VM.

However, to utilize this special plugin multiple things have to be true:
- The storage (Storage) that will host the ZFS/ISCS must allow ssh
- The Storage must allow ssh as root
- The Storage must be ZFS based at its core
- The Storage must allow direct ZFS manipulations by common ZFS toolset
- The Storage must have supported iSCSI daemon
- The Storage must allow iSCSI configuration manipulation by common iSCSI toolset

AFAIK, TrueNAS may fit all of the above but not with the default ZFS/iSCSI plugin that ships with Proxmox. You can search github, there are several modifications there that claim to work.

If you plug "zfs iscsi" and optionally "truenas" into Forum search, you will find many discussions about this:
https://forum.proxmox.com/threads/proxmox-8-1-with-synology-uc3200-iscsi.141057/#post-631478
https://forum.proxmox.com/threads/zfs-over-iscsi.140546/#post-628686
https://forum.proxmox.com/threads/h...th-zfs-to-proxmox-cluster.140254/#post-627364
https://forum.proxmox.com/threads/enterprise-alternatives-to-vmware.140174/#post-627842

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

Thank you, I'll give those a read. So far I haven't even managed to get LVM over iSCSI working so there's a earning curve

 

[Marco2G]

I gotta say first steps aren't all that heart-wearming. I'm trying to do ZFS over iSCSI according to https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

And it doesn't work. Certainly not going to introduce Proxmox to my boss for now .

 

[bbgeek17]

As I said previously, ZFS/iSCSI with TrueNAS will likely require a customized storage Plugin that is NOT shipped with Proxmox distribution.
You have to find it, install it, configure it, and support it outside of normal PVE support channels.

While I am aware that some forum members use ZFS/iSCSI in lab and even small scale production, it would not be my recommendation to use it in larger deployments unless you understand the technology end to end, and ready to troubleshoot it on your own.

If you need to present to your boss out of the box, supportable solutions, you will likely need to look elsewhere for storage.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

I have understood that part. The problem lies in host key access refusing to work as described by the wiki.

 

[bbgeek17]

I have understood that part. The problem lies in host key access refusing to work as described by the wiki.

That sounds like an issue of general Linux/SSH management, rather than Proxmox Hypervisor.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Marco2G]

Yes, absolutely. My comment was more towards the expectation of seamlessness we have from vSphere that I already find will need a bit of adjustment from my side with Proxmox.

Don't get me wrong, I wish that Proxmox rses to be avalid contender for Vmware and I am absolutely sure I will get it to run my homelab as well as ESXi did, eventually. But truth of the matter is open source has always been painful, is still so and will continue to be so for quite some time.

(Not that roadcom, Microsoft and Apple have customer satisfaction in their mission statements )

 

[bbgeek17]

Your sentiment is not unreasonable and often happens in all fields and against all products.

There are different expectations between homelab and business environment, budget and no-budget, enthusiast and 9-5 sysadmin with a task at hand and bonus on the line. There is definitely an adjustment period when switching products, especially as abruptly as it is these days.

As I said, you have a choice between build it yourself and support/license via developer or partners. It all depends on your needs and means.

Good luck with whatever path you pick

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox