[TUTORIAL] PVE 7.x Cluster Setup of shared LVM/LV with MSA2040 SAS [partial howto]

Hello,

With " Before=pve-guests.service" the node startup work fine.

But when I shutdown/restart the node, the HA VMs do not stop and the gfs2 filesystem cannot umount... (fuser say filesystem in use by kvm).

I tested the command "pvesh create /nodes/localhost/stopall" used by the "pve-guests.service" to stop, HA VM do not stop.

What is the service who start/stop HA VMs ?

I need to run "lvmshare.service" before the VMs startup (pve-guests.service) and the HA VMs startup.

Best regards.

Francis
 
Hello,

I changing "Before=pve-guests.service" by " Before=pve-ha-lrm.service pve-guests.service" all work fine I can start/stop the nodes.

[Unit]
Description=LVM locking LVs and mount LVs start and stop
Documentation=man:lvmlockd(8)
After=lvmlocks.service lvmlockd.service sanlock.service dlm.service
Before=pve-ha-lrm.service pve-guests.service

[Service]
Type=oneshot
RemainAfterExit=yes

# start lockspaces LVs and mount LVs
ExecStart=/usr/bin/bash -c "/usr/sbin/vgs --noheadings -o name -S vg_shared=yes | xargs /usr/sbin/lvchange -asy"
ExecStart=/usr/bin/bash -c "/usr/sbin/lvs --noheadings -o lv_path -S vg_shared=yes | xargs mount"

# stop lockspaces LVs after umount LVs
ExecStop=/usr/bin/bash -c "/usr/sbin/lvs --noheadings -o lv_path -S vg_shared=yes | xargs umount"
ExecStop=/usr/bin/bash -c "/usr/sbin/vgs --noheadings -o name -S vg_shared=yes | xargs /usr/sbin/lvchange -an"

[Install]
WantedBy=multi-user.target

Best regards.

Francis
 
@Glowsome questions for you

per your experiment , Can we have to link enable in proxmox that will not cause issue to DLM / GFS2 Services ?
i think i had read something in the past that corosync didnt supported 2 network simultanously

other question
if it cant , do you think the option for Qdevice can do the job as it will maintain the pvecm service alive ?
 
@Glowsome questions for you

per your experiment , Can we have to link enable in proxmox that will not cause issue to DLM / GFS2 Services ?
i think i had read something in the past that corosync didnt supported 2 network simultanously

other question
if it cant , do you think the option for Qdevice can do the job as it will maintain the pvecm service alive ?
My setup is running with a single (well bonded) interface in active-backup config network.
I did not see a/the need to work with a separate network just for cluster information.
 
A little thing i encountered in my setup :
After a reboot (needed after upgrading PVE7 -> PVE8) i got errors from DLM, and it basically meant that it was getting blocked (sk_err 110/0 was seen on consoles).

Then i remembered i had tightened security by turning on firewalls on the individual nodes, and only allowing a strict set of services like 2 months ago.

Apparently when you turn on a/the firewall it will not kill already established connections, so everything functioned up untill i rebooted

So lesson learned .. IF you enable the firewall on a clusternode, make sure you allow the port DLM uses (tcp/21064) !!!

- Glowsome
 
Dear @Glowsome,
why do you implement it so complicated? Would not the use of OCFS2 on the Block storage in combination with DIR Storage do the trick. Or do u need the speed gain from lvm?
 
Dear @Glowsome,
why do you implement it so complicated? Would not the use of OCFS2 on the Block storage in combination with DIR Storage do the trick. Or do u need the speed gain from lvm?
This whole thing started as a project to get around limitations i encountered.
and as a whole, the usage of OCFS2 is oracle, and after the took over MySQL, i dread being locked in by using this.

Due to the nature it grew.. and grew ... and maybe yes its turned into a beast .. but still all in all i'm not after something like a speed-gain.
Concider it a proof of concept, and its still surviving over and over again fulfilling my needs since i stepped in using PVE 5.x
Latest is i migrated all 4 nodes to PVE 8.x.

- Glowsome
 
Last edited:
Hello,

About the "sleep 10" added int the unitfile "lvmlockd.service" you do not have to modify the file.

You can create a directory "/etc/systemd/lvmlockd.service.d/" and create a file "sleep.conf" (exemple) with the content

[Service]
ExecStartPre=/usr/bin/sleep 10

or you can create a directory "/etc/systemd/dlm.service.d/" and create a file "sleep.conf"

[Service]
ExecStartPost=/usr/bin/sleep 10

An update can overwrite the file "lvmlockd.service" or "dlm.service" no problem.

Best regards.

Francis
Hi, might be a late reply on this, but if i read correctly then dropping an addition in:
/etc/systemd/lvmlockd.service.d/
This means you are not adding, but actually overriding.
(for reference) see :
https://access.redhat.com/documenta...tion_assembly_working-with-systemd-unit-files

So the correct place to drop these additions (IMHO) would be in:
Code:
/etc/systemd/system/servicename.service.d/*

Where in my case, the additions like the sleep timings, aswell as the additional dependancies introduced like After=
Should be dropped there in contrary of your own suggested context.

We want to add conditions or , not override existing ones.
 
Hello Glowsome

Yes sorry the correct path for the file is "/etc/systemd/system/lvmlockd.service.d/".

Where in my case, the additions like the sleep timings, aswell as the additional dependancies introduced like After=
Should be dropped there in contrary of your own suggested context.

I do not understand.

Best regards.

Francis
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!