shared storage between 2 nodes within a cluster over common SAS disk

François Vivet

New Member
Sep 16, 2022
11
0
1
Hi, I'm wondering how to use a DELL ME5024 attached via SAS cards to two members of a PROXMOX VE cluster. The idea is to offer a direct access to the VM disks to the 2 nodes and to reduce the migration time when one of the nodes fells down. Any help would be appreciate.
My cluster is in version 7.3-4
 
  • Like
Reactions: François Vivet
The table in https://pve.proxmox.com/pve-docs/chapter-pvesm.html states "Shared: no". But the footnote reads:

"It is possible to use LVM on top of an iSCSI or FC-based storage. That way you get a shared LVM storage."

Please note that this approach is not capable of creating snapshots...
Effectively I was abble to tick the share option for the LVM volume I created on one of the node. My pb now is to make this LVM storage active on the second node when the first on which I created the LVM volume is off. I made some tests and the 2nde node was not abble to activate this LVM storage. It seems correct as the second node even not have a PV which point to this "shared" storage. So now I'm wondering what are the steps I have to follow to reach my aim which is to have a LVM shared storage which is active on the first node and then switches to the second when the first is in fault.
 
The table in https://pve.proxmox.com/pve-docs/chapter-pvesm.html states "Shared: no". But the footnote reads:

"It is possible to use LVM on top of an iSCSI or FC-based storage. That way you get a shared LVM storage."

Please note that this approach is not capable of creating snapshots...
It seems that it does not work for SAS-based storage: after that I created a LVM-thick storage on node 1, allowing it at he cluster level I find it on node 2 -in the un-activated state). My pb is that I don't see how to activate it on node 2. I understand that a LVM storage can not be activated on two nodes at a time but I have no idea on how to make it activated when node 1 is off.
 
this only works if your nodes are clustered. since you (apparently) have only two nodes, you will need to add a third (quorum) node to accomplish this. see https://pve.proxmox.com/wiki/Cluster_Manager on how to set up a qdevice on a third node- it doesnt have to participate in the shared resource serving, just in establishing quorum.
Actually my nodes are in a cluster which has a total of 3 nodes, the third one being there only for the quorum.
 
Actually my nodes are in a cluster which has a total of 3 nodes, the third one being there only for the quorum.
then it will work as advertised :)

Its just that it's not a common solution within the proxmox use cases, since sas is only two channels and a min cluster size is three. logically, the system doesn't really differentiate from iscsi and sas as they are both SBP (scsi block protocol.)
I understand that a LVM storage can not be activated on two nodes at a time
Since your storage is attached to a cluster aware environment, not only you can, but you must. vgchange -ay should do the trick; if it doesnt activate at boot you may need to put that in as a post boot script or service.
 
then it will work as advertised :)

Its just that it's not a common solution within the proxmox use cases, since sas is only two channels and a min cluster size is three. logically, the system doesn't really differentiate from iscsi and sas as they are both SBP (scsi block protocol.)

Since your storage is attached to a cluster aware environment, not only you can, but you must. vgchange -ay should do the trick; if it doesnt activate at boot you may need to put that in as a post boot script or service.
Something that I do not totally explain happened: I created a new VM on the 2nd node of my proxomox cluster allocating to this VM a hard-drive on the SAS storage (I just wanted to try and see what would happen) shared between the 2 nodes but only active on the 1rst node at this time. The LVM VG became magically also active on the 2nd node. So right now the shared SAS storage is active on the two nodes. It seems that ticking the shared option on the storage declaration at the cluster level has no effect in a sens or another on the way the sharing works.
 
just to clarify, the 'shared' checkbox in the gui does not 'share' anything, it tells the PVE stack that the underlying storage *is* shared, so that e.g. for live-migration no volumes must be copied
As for LVM, PVE normally takes care about activation/deactivation by itself, so there should nothing to be done manually (and AFAIU the thread this was the case?)
 
just to clarify, the 'shared' checkbox in the gui does not 'share' anything, it tells the PVE stack that the underlying storage *is* shared, so that e.g. for live-migration no volumes must be copied
As for LVM, PVE normally takes care about activation/deactivation by itself, so there should nothing to be done manually (and AFAIU the thread this was the case?)
Regarding LVM on a SAS attached storage between 2 nodes (in a cluster of 3 nodes), as I said I created a VG on node 1 and allowed it for 2 nodes (node 1 and node 2) through the GUI "cluster" menu. At this step the VG was seen on both nodes but active only on node 1 (as it was clearly mentionned in the GUI). This VG became active for both nodes when I tried to create a VM on the node 2 and declared a hard-drive on the shared LVM: it seems that the creation of a LV (the VM hard-drive) on the shared LVM storage activated the VG on node 2. To sumerize it was partially automatic ;)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!