Storage - iscsi configuration

[jwang]

I'm configuring Proxmox (PM) 8.2.2 with TrueNas (TNs) Scale 24.x It seems PM offers many options here and so I'm trying to find my way to what is recommended as well as potentially a sensible storage configuration.

1. I'm not sure which storage option to use with TNs iscsi. Storage on PM could be just "iscsi" (what I've configured now) or "zfs over iscsi." I tried to make zfs over iscsi work (there is a patch to do that) and failed - then noticed all/some aspect of this is "legacy" in the user-guide.
Is straight iscsi recommended here?

2. It seems I could configure storage on the TNs side so each zvol houses one vm. This seems to get very cluttered fast on the PM side. Alternatively, a large zvol could be shared with LVM on PM between multiple VMs. This seems less cluttered and recommended elsewhere in the forums.

However, I see the PM guide states,

• On block level storage, the underlying storage layer provides block devices (similar to actual disks) which are used for disk images. Functionality like snapshots are provided by the storage layer itself. Examples: ZFS, Ceph RBD, thin LVM, …

So it seems with a large zvol I give up granular control over snapshots at the VM level?
Just to add to my confusion - each vm on the PM side has a snapshot option.

What might be a sound approach here and how are snapshots best managed?

I may have missed something here? I'd appreciate a bit of input particularly on how to bring the storage in from TNs.
Thanks!

 

[bbgeek17]

I'm not sure which storage option to use with TNs iscsi. Storage on PM could be just "iscsi" (what I've configured now)

This is the supported option. It involves an industry standard protocol with minimum to no custom wrapping.

"zfs over iscsi."

This is a PVE specific storage scheme that works out of the box only with specific list of products. As you discovered, an enhanced custom plugin is required for TrueNAS. This plugin is not supported by PVE, and any questions about its functionality should be directed to the plugin's support channel.

So it seems with a large zvol I give up granular control over snapshots at the VM level?

regardless of what the backend (TrueNAS) is using for internal storage management (ZFS, XFS, etc), PVE will see the exported LUNs as raw blocks. You will need to pass that raw block storage to VM. There are a few options:
- In PVE place LVM on top of the raw iSCSI LUN. This will allow you to keep the number of LUNs to a minimum. PVE will slice and dice the LVM as needed. There is no snapshot support
- pass-through the iSCSI LUN to VM directly. You will have as many iSCSI LUNs off your TrueNAS as you have VM disks in PVE. There is no snapshot support with this option. You can implement manual snapshots directly on TrueNAS without PVE awareness.
- implement ZFS over iSCSI. If you can get it working you will have (as an example) 10 ZFS slices/volumes on your TrueNAS, exported as 10 raw iSCSI LUNs, matching 10 PVE disk images (virtual disks). Snapshot support is integrated in PVE but is executed on TrueNAS on a per ZFS slice basis.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jwang]

This is the supported option. It involves an industry standard protocol with minimum to no custom wrapping.

This is a PVE specific storage scheme that works out of the box only with specific list of products. As you discovered, an enhanced custom plugin is required for TrueNAS. This plugin is not supported by PVE, and any questions about its functionality should be directed to the plugin's support channel.

regardless of what the backend (TrueNAS) is using for internal storage management (ZFS, XFS, etc), PVE will see the exported LUNs as raw blocks. You will need to pass that raw block storage to VM. There are a few options:
- In PVE place LVM on top of the raw iSCSI LUN. This will allow you to keep the number of LUNs to a minimum. PVE will slice and dice the LVM as needed. There is no snapshot support
- pass-through the iSCSI LUN to VM directly. You will have as many iSCSI LUNs off your TrueNAS as you have VM disks in PVE. There is no snapshot support with this option. You can implement manual snapshots directly on TrueNAS without PVE awareness.
- implement ZFS over iSCSI. If you can get it working you will have (as an example) 10 ZFS slices/volumes on your TrueNAS, exported as 10 raw iSCSI LUNs, matching 10 PVE disk images (virtual disks). Snapshot support is integrated in PVE but is executed on TrueNAS on a per ZFS slice basis.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for laying out the 3 options. The iSCSI has been relatively straightforward to setup. I couldn't get the plugin to work (I'm first to admit I'm not the most qualified) and the support was not very active. The plugin makes me nervous about security (involving the GUI, privileged user and password).

So this narrows me to two options. 1. Using LVM ontop of the LUN or 2. one LUN one VM.
Just to clarify on 1:

- In PVE place LVM on top of the raw iSCSI LUN. This will allow you to keep the number of LUNs to a minimum. PVE will slice and dice the LVM as needed. There is no snapshot support

This is my current direction. Is this "no snapshots" on the PM side? Presumably I could do this on the TN side although it will cover the whole LUN (several VMs)?
Using 2. No snapshots on PM but on TN - and per VM (so more granular).

Thanks for the input so far. I just want to make sure I'm not missing something here.