Proxmox + TrusNAS ISCSI

[rigorm]

Hi,
I'm currently looking at building a 4 nodes proxmox and a TrueNAS backend.

Vm volumes will be on shared using ISCSI on truNAS.

My question is : Whats the best practice for the shared volumes accross multiple nodes ?

1. Sharing one big LUN and put LVM on top and have all volumes on LVM.
   1. Is there a problem with this ?
   2. I'm I going to have LVM problems if shared across all nodes ?
2. or is it better to skip LVM and simply use smaller LUNs and have 1 per VM ?

I'm looking at option #1 but would like some inputs

thanks

 

[Dunuin]

I think you will need to use multipath.

 

[rigorm]

I think you will need to use multipath.

I though that multipath was when you have multiple switches paths connected to the SAN and hosts ?!

This is not my case btw

 

[mmenaz]

Hi,
I'm currently looking at building a 4 nodes proxmox and a TrueNAS backend.

Vm volumes will be on shared using ISCSI on truNAS.

My question is : Whats the best practice for the shared volumes accross multiple nodes ?

1. Sharing one big LUN and put LVM on top and have all volumes on LVM.
   1. Is there a problem with this ?
   2. I'm I going to have LVM problems if shared across all nodes ?
2. or is it better to skip LVM and simply use smaller LUNs and have 1 per VM ?

I'm looking at option #1 but would like some inputs

thanks

Never used it and I could say something meaningfulness, but why not "ZFS over SCSI"?
https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI

 

[bbgeek17]

Option 1 is fine and is officially supported by Proxmox. There are restrictions as listed here: https://pve.proxmox.com/wiki/Storage

I.e.: LVM must be in thick mode, snapshots are not supported.
Many people use this approach in the community. The trick with LVM in shared storage is that only one host/vm must access the slice at any given time. If multiple hosts try to access the same slice you will have data corruption. So there must be an arbitrator of access. This is what Proxmox manager does - it only activates the slice where specific VM is running. It works reasonably well.

Option 2 is a much more manually involved process, but it also works. If you build good automation around it it will be functional. There is still no snapshot support via Proxmox, but you can probably snapshot individual LUNs on the backend if you want.

Option 3 as @mmenaz mention is there but its not plug-and-play. The ZFS/iSCSI relies on ability of Proxmox to: SSH to storage, manipulate ZFS pool, manipulate iSCSI daemon. Specifically it expects iSCSI service from a narrowly defined set of services. As far as I know TrueNAS no longer runs one of those. If you look around github you may find some projects trying to adapt to new TrueNAS packaging. How well it works is up to you find out.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox