FC SAN with Proxmox cluster

H

hanse

Member
Jul 25, 2021
11
2
8
38
Hi all. I am a total newbie when it comes to SAN/iSCSI/FC. I have read just about every post or thread I could find on the matter, but I am guessing I am too stupid to grasp what to do here. (And not scared to admit that).

We have an enclosure with 12 nodes that I want to all connect to a SAN storage device. So the first node I have deployed Proxmox PVE on, and have installed multipath on. But from here I am stuck. From below you can see we have a lot of drives in the SAN:

Code: 
ls /dev/sd
Display all 188 possibilities? (y or n)
sda    sdah   sdaq   sday   sdbf   sdbo   sdbw   sdce   sdf    sdo    sdx
sda1   sdah1  sdaq1  sday1  sdbf1  sdbo1  sdbw1  sdce1  sdf1   sdo1   sdx1
sda2   sdai   sdar   sdaz   sdbg   sdbp   sdbx   sdcf   sdg    sdp    sdx2
sda3   sdai1  sdar1  sdaz1  sdbg1  sdbp1  sdbx1  sdcf1  sdg1   sdp1   sdx3
sdaa   sdaj   sdas   sdb    sdbh   sdbp2  sdby   sdcg   sdh    sdq    sdy
sdaa1  sdaj1  sdas1  sdb1   sdbh1  sdbp3  sdby1  sdcg1  sdh1   sdq1   sdy1
sdab   sdak   sdat   sdb2   sdbi   sdbq   sdbz   sdch   sdi    sdr    sdz
sdab1  sdak1  sdat1  sdb3   sdbi1  sdbq1  sdbz1  sdch1  sdi1   sdr1   sdz1
sdac   sdal   sdat2  sdba   sdbj   sdbr   sdc    sdci   sdj    sds  
sdac1  sdal1  sdat3  sdba1  sdbj1  sdbr1  sdc1   sdci1  sdj1   sds1  
sdad   sdam   sdau   sdbb   sdbk   sdbs   sdca   sdcj   sdk    sdt  
sdad1  sdam1  sdau1  sdbb1  sdbk1  sdbs1  sdca1  sdcj1  sdk1   sdt1  
sdae   sdan   sdav   sdbc   sdbl   sdbt   sdcb   sdck   sdl    sdu  
sdae1  sdan1  sdav1  sdbc1  sdbl1  sdbt1  sdcb1  sdck1  sdl1   sdu1  
sdaf   sdao   sdaw   sdbd   sdbm   sdbu   sdcc   sdd    sdm    sdv  
sdaf1  sdao1  sdaw1  sdbd1  sdbm1  sdbu1  sdcc1  sdd1   sdm1   sdv1  
sdag   sdap   sdax   sdbe   sdbn   sdbv   sdcd   sde    sdn    sdw  
sdag1  sdap1  sdax1  sdbe1  sdbn1  sdbv1  sdcd1  sde1   sdn1   sdw1

From our storage guy I gathered that on the SAN we have a total of 22 Volumes being exposed.
So question 1 is, is there a way to get all of the wwid's of all these disks to add the wwid to the multipath config? Or am I completely off track already?

EDIT: By using a for loop and a lot of manual work, I was able to determine that there are indeed 22 unique wwid output from /lib/udev/scsi_id -g -u -d /dev/sdX commands. So I guess these 22 ids correspond to the 22 volumes exposed.

I would really appreciate any help I can get with this.
Regards
Hans
 
Last edited:
While there may be a valid reason to program WWID individually, most likely you can get away with using a wildcard. Keep in mind that PVE runs on Debian and so this low level storage configuration is standard and not PVE specific. If you expand your search, for example google "multipath command in linux with examples" then this link comes across: https://www.thegeekdiary.com/understanding-linux-multipath-using-dm-multipath/

The more important part is - what do you want to achieve with your setup? At first glance the storage layout that was provided to you is not optimal for most common configurations.
Is the goal to have shared storage for your PVE cluster? Have you decided how you will arbitrate shared access? https://en.wikipedia.org/wiki/Clustered_file_system#Examples

Perhaps each disk will be dedicated to specific VM? Or are you going to have 22 LVM-thick datastores? I would sit down with your storage guy and really map out what the environment goals are and what it should look like.

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
So I was tasked with evaluating Proxmox in our environment to replace VMWare.

To give you background. This SAN has a combination of different disk sizes and speeds. Similar ones were grouped together to create the exposed volumes. In a traditional vmware way these volumes were exposed/configured in vmware.

Our idea was to use it in a similar fashion that with VMWare, where we have it "shared" so I can migrate vms and cts between different nodes inside the cluster. (I know that inter-cluster migration is not possible yet, but indeed a feature request). In other words, if I want to create a vm, I select the storage I want it to live on (DB on SSD volumes, long term storage on spinning rust etc) and be able to move machines between hosts in the cluster without issue.
So no single disk would need to be accessed by just a single VM.

Lastly, yes, if it means I have to create 22 thick LVM pools in order to have backup, snapshot and migration capabilities, then that is what I will do, as this will be a once off? I saw that the storage file will automatically be shared between nodes so that we don't have to configure it for each node.
I will look into the wildcard method and see if I can make sense of it all. Thank you @bbgeek17
 
Without knowing low level details about your SAN, I am going to assume a few things, chiefly that your SAN has built-in disk protection (RAID). That means the LUNs that you present to clients (PVE nodes) are already protected from disk failure.

A very rough equivalent of VMWare VMFS datastore would be Proxmox Ceph. And I mean it at 10000 foot view, because they are completely different things under the cover. There is no VMFS like technology that comes built-into Proxmox.

One thing you should understand is the table here : https://pve.proxmox.com/wiki/Storage.
After you filter by shared=yes and throw away all incompatible technologies, ie ZFS/iSCSI. You are left with:
- GlusterFS (not recommended for centralized SAN)
- Ceph (not recommended for centralized SAN)
- LVM (thick) - no snapshot support from Proxmox, you could snapshot LUNs on SAN but its most likely not what you want.
- One of not listed Clustered Filesystems with no direct support from PVE (https://en.wikipedia.org/wiki/Clustered_file_system#Examples)

My advice is:
a) Reduce the number of LUNs to as little as possible. Tier1=1Lun, Tier2=1Lun, etc. Your storage vendor may recommend more than 1 LUN for performance reasons, but then you are introducing potential cross-storage migration which is not online. Its not the end of the world, but I would make the LUNs as large as possible.
b) Use Clustered Filesystem - pick one and find a guide on how to install it on Debian. Use Qcow VM image from Proxmox for snapshot support

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Without knowing low level details about your SAN, I am going to assume a few things, chiefly that your SAN has built-in disk protection (RAID). That means the LUNs that you present to clients (PVE nodes) are already protected from disk failure.

A very rough equivalent of VMWare VMFS datastore would be Proxmox Ceph. And I mean it at 10000 foot view, because they are completely different things under the cover. There is no VMFS like technology that comes built-into Proxmox.

One thing you should understand is the table here : https://pve.proxmox.com/wiki/Storage.
After you filter by shared=yes and throw away all incompatible technologies, ie ZFS/iSCSI. You are left with:
- GlusterFS (not recommended for centralized SAN)
- Ceph (not recommended for centralized SAN)
- LVM (thick) - no snapshot support from Proxmox, you could snapshot LUNs on SAN but its most likely not what you want.
- One of not listed Clustered Filesystems with no direct support from PVE (https://en.wikipedia.org/wiki/Clustered_file_system#Examples)

My advice is:
a) Reduce the number of LUNs to as little as possible. Tier1=1Lun, Tier2=1Lun, etc. Your storage vendor may recommend more than 1 LUN for performance reasons, but then you are introducing potential cross-storage migration which is not online. Its not the end of the world, but I would make the LUNs as large as possible.
b) Use Clustered Filesystem - pick one and find a guide on how to install it on Debian. Use Qcow VM image from Proxmox for snapshot support

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Thank you @bbgeek17 for explaining this so properly. You are correct, snapshot is a must-have. And not on SAN. So I guess this whole operation is a bit futile. But I am sure your proper explanation will definitely help other newcomers and minimally informed others like myself.
 
hanse said:
Thank you @bbgeek17 for explaining this so properly. You are correct, snapshot is a must-have. And not on SAN. So I guess this whole operation is a bit futile. But I am sure your proper explanation will definitely help other newcomers and minimally informed others like myself.
Click to expand...

I wouldn't say its futile, it just requires more work than simple Proxmox cluster installation. You can take a LUN (I'd start with a single one instead of 22), deploy and configure Oracle Cluster File System (OCFS) on each of the cluster nodes and then use Proxmox directory storage to store images as QCOW. The qcow format buys you snapshot support.

Of course only you can decide if any of this is worth the effort.

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I work oder 15 Years with VMware.
Tue easyiest Way for VMware Guys are, use thin LVM or format your LUNs with ZFS.
 
For Design new Hardware, i recommend HCI with Ceph.
 
OK, i have read any things, ZFS formated LUNs are not shareable. :rolleyes:
Only LVM or GlusterFS is possible, but GlusterFS is very tricky and not recommended.
 
GFS2 can also be used with the kernel, but is not offered in the GUI. GFS2 is also snapshot capable, but I don't know if Proxmox supports it.
 
SkyDiver79 said:
Which Protocol use Blockbridge?
Click to expand...

For Proxmox, our current recommendation is iSCSI. We build all-flash datacenter solutions primarily used by service providers, where performance, availability, and automation are essential.

We've been helping folks run Proxmox on shared storage for quite a while now. Initially, this was using LVM. More recently, we released a storage driver that natively integrates Proxmox API primitives with our API and host orchestration tools. In effect, the integration automates fabric management, security, high availability, multipathing, encryption, data reduction, etc.

From a storage management perspective, there is a 1 to 1 mapping between storage concepts. For example:
  • PVE disk is a BB disk.
  • PVE disk create == BB virtual disk create
  • PVE disk snapshot == BB disk snapshot
  • PVE linked clone == BB thin clone
  • PVE snap rollback == BB snap rollback
You can find some in-depth information in our knowledge base: https://kb.blockbridge.com/guide/proxmox/

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom