Proxmox + TrusNAS ISCSI

rigorm

Renowned Member
Feb 6, 2011
17
0
68
Hi,
I'm currently looking at building a 4 nodes proxmox and a TrueNAS backend.

Vm volumes will be on shared using ISCSI on truNAS.

My question is : Whats the best practice for the shared volumes accross multiple nodes ?
  1. Sharing one big LUN and put LVM on top and have all volumes on LVM.
    1. Is there a problem with this ?
    2. I'm I going to have LVM problems if shared across all nodes ?
  2. or is it better to skip LVM and simply use smaller LUNs and have 1 per VM ?
I'm looking at option #1 but would like some inputs

thanks :)
 
Hi,
I'm currently looking at building a 4 nodes proxmox and a TrueNAS backend.

Vm volumes will be on shared using ISCSI on truNAS.

My question is : Whats the best practice for the shared volumes accross multiple nodes ?
  1. Sharing one big LUN and put LVM on top and have all volumes on LVM.
    1. Is there a problem with this ?
    2. I'm I going to have LVM problems if shared across all nodes ?
  2. or is it better to skip LVM and simply use smaller LUNs and have 1 per VM ?
I'm looking at option #1 but would like some inputs

thanks :)
Never used it and I could say something meaningfulness, but why not "ZFS over SCSI"?
https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI
 
Option 1 is fine and is officially supported by Proxmox. There are restrictions as listed here: https://pve.proxmox.com/wiki/Storage

I.e.: LVM must be in thick mode, snapshots are not supported.
Many people use this approach in the community. The trick with LVM in shared storage is that only one host/vm must access the slice at any given time. If multiple hosts try to access the same slice you will have data corruption. So there must be an arbitrator of access. This is what Proxmox manager does - it only activates the slice where specific VM is running. It works reasonably well.

Option 2 is a much more manually involved process, but it also works. If you build good automation around it it will be functional. There is still no snapshot support via Proxmox, but you can probably snapshot individual LUNs on the backend if you want.

Option 3 as @mmenaz mention is there but its not plug-and-play. The ZFS/iSCSI relies on ability of Proxmox to: SSH to storage, manipulate ZFS pool, manipulate iSCSI daemon. Specifically it expects iSCSI service from a narrowly defined set of services. As far as I know TrueNAS no longer runs one of those. If you look around github you may find some projects trying to adapt to new TrueNAS packaging. How well it works is up to you find out.


Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!