Proxmox, 6 node cluster with FC - Emulex HBA cards - SAN Volumes

[kiranch97]

Hi Support,

We are planning to setup a six node Proxmox env with SAN connectivity (FC - Emulex HBA cards), I couldnt find online documentation on how to add the block SAN volume into cluster nodes. Can you please help share procedure, appreciated for your support.

 

[bbgeek17]

You can subdivide the setup into logical steps/phases:
1) physical connectivity
2) drivers/firmware/etc
3) lun presentation
4) multipath if any
5) volume manager or file system layout
6) PVE setup

Phases 1-4 are completely non-PVE related. You need to find your SAN vendor documentation on how to connect said SAN to Debian 11. There may or may not be Kernel specific steps.

Step 5 is somewhat PVE specific as you need to decide how you will logically present storage to PVE. Your choices are:
a) thick LVM - no snapshot support.
b) Cluster Aware File system - none provided by PVE, you will need to install/configure and support it on your own
c) LUN per virtual disk approach, ie direct LUN access. May be too labor intensive, depending on SAN/setup.

Step 6 is when you finally start configuring PVE/Proxmox itself. This is where you define the context of /etc/pve/storage.conf or how the VMs will be seeing the LUNs.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kiranch97]

Hi bbgeek17, Thanks alot, I was seeing

referring to #post-516198, listing the approach for the FC SAN connectivity and Volumes assigning to cluster nodes:

1. Base installation on all 6 nodes
2. Installing multipath
command:
apt-get install multipath-tools
3. Setting up cluster
i. pvecm create
ii. pvecm add

4. Creating volumegroup for SAN volumes on node 1:
i. pvcreate /dev/mapper/...
ii. vgcreate <volumegroup> /dev/mapper

5. Adding storage as LVM in proxmox-web-gui (with option "shared" checked)
6. {not clear adding same shared volume in rest of nodes}

Can you suggest on point 6, will this approach is thick-LVM and supported by proxmox ?

 

[bbgeek17]

the "shared" checkmark will properly provision the LVM as shared storage compatible.
You wont need to duplicate PVE storage configuration across nodes in cluster. If you have properly configured the cluster, the configuration will be distributed automatically.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kiranch97]

Awesome, Thanks @bbgeek17 !! Will attempt to configure.

will this approach is supported by proxmox support incase if we open any direct support ticket?
will this approach is called thin-LVM or thick LVM, wanted to whether vms provisioning would be thin provisioned so that i cannot consume storage space at backend storage when there is no much disk usage inside VM?

Please advise.

 

[bbgeek17]

will this approach is supported by proxmox support incase if we open any direct support ticket?

great question to ask Proxmox support/sales team office@proxmox.com. Probably also a good idea to ask your SAN vendor whether will support you in this setup as well.

thin-LVM or thick LVM

you cannot use thin provisioned LVM with shared storage - you will have data corruption.
thick provisioned LVM allocates the entire space out of the pool to a specific LVM slice as you create your virtual disks, based on this disk size. There is no backend/LVM snapshot support with thick LVM.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

this may be of use to you:
https://forum.proxmox.com/threads/p...-lvm-lv-with-msa2040-sas-partial-howto.57536/

 

[kiranch97]

great question to ask Proxmox support/sales team office@proxmox.com. Probably also a good idea to ask your SAN vendor whether will support you in this setup as well.

you cannot use thin provisioned LVM with shared storage - you will have data corruption.
thick provisioned LVM allocates the entire space out of the pool to a specific LVM slice as you create your virtual disks, based on this disk size. There is no backend/LVM snapshot support with thick LVM.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi @bbgeek17 from SAN vendor there are no challenges on the Volume allocation to any hypervisior, it assume its how the hypervisior handle the SAN disk.

once we make the SAN disk allocation and mark it as LVM shared, can you please advise in case when the case arrive to expand one existing SAN disk, can you please assist how we expand it from proxmox level.

 

[bbgeek17]

can you please advise in case when the case arrive to expand one existing SAN disk, can you please assist how we expand it from proxmox level.

I imagine it would be standard: expand your physical disk, expand the underlying LVM layer PV/VG.
While you are in implementation phase - why not just try it to be sure?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox