[TUTORIAL] PVE 7.x Cluster Setup of shared LVM/LV with MSA2040 SAS [partial howto]

[FrancisS]

Hello,

With " Before=pve-guests.service" the node startup work fine.

But when I shutdown/restart the node, the HA VMs do not stop and the gfs2 filesystem cannot umount... (fuser say filesystem in use by kvm).

I tested the command "pvesh create /nodes/localhost/stopall" used by the "pve-guests.service" to stop, HA VM do not stop.

What is the service who start/stop HA VMs ?

I need to run "lvmshare.service" before the VMs startup (pve-guests.service) and the HA VMs startup.

Best regards.

Francis

 

[FrancisS]

Hello,

I changing "Before=pve-guests.service" by " Before=pve-ha-lrm.service pve-guests.service" all work fine I can start/stop the nodes.

[Unit]
Description=LVM locking LVs and mount LVs start and stop
Documentation=man:lvmlockd(8)
After=lvmlocks.service lvmlockd.service sanlock.service dlm.service
Before=pve-ha-lrm.service pve-guests.service

[Service]
Type=oneshot
RemainAfterExit=yes

# start lockspaces LVs and mount LVs
ExecStart=/usr/bin/bash -c "/usr/sbin/vgs --noheadings -o name -S vg_shared=yes | xargs /usr/sbin/lvchange -asy"
ExecStart=/usr/bin/bash -c "/usr/sbin/lvs --noheadings -o lv_path -S vg_shared=yes | xargs mount"

# stop lockspaces LVs after umount LVs
ExecStop=/usr/bin/bash -c "/usr/sbin/lvs --noheadings -o lv_path -S vg_shared=yes | xargs umount"
ExecStop=/usr/bin/bash -c "/usr/sbin/vgs --noheadings -o name -S vg_shared=yes | xargs /usr/sbin/lvchange -an"

[Install]
WantedBy=multi-user.target

Best regards.

Francis

 

[DC-CA1]

@Glowsome questions for you

per your experiment , Can we have to link enable in proxmox that will not cause issue to DLM / GFS2 Services ?
i think i had read something in the past that corosync didnt supported 2 network simultanously

other question
if it cant , do you think the option for Qdevice can do the job as it will maintain the pvecm service alive ?

 

[Glowsome]

@Glowsome questions for you

per your experiment , Can we have to link enable in proxmox that will not cause issue to DLM / GFS2 Services ?
i think i had read something in the past that corosync didnt supported 2 network simultanously

other question
if it cant , do you think the option for Qdevice can do the job as it will maintain the pvecm service alive ?

My setup is running with a single (well bonded) interface in active-backup config network.
I did not see a/the need to work with a separate network just for cluster information.

 

[Glowsome]

A little thing i encountered in my setup :
After a reboot (needed after upgrading PVE7 -> PVE8) i got errors from DLM, and it basically meant that it was getting blocked (sk_err 110/0 was seen on consoles).

Then i remembered i had tightened security by turning on firewalls on the individual nodes, and only allowing a strict set of services like 2 months ago.

Apparently when you turn on a/the firewall it will not kill already established connections, so everything functioned up untill i rebooted

So lesson learned .. IF you enable the firewall on a clusternode, make sure you allow the port DLM uses (tcp/21064) !!!

- Glowsome

 

[floh8]

Dear @Glowsome,
why do you implement it so complicated? Would not the use of OCFS2 on the Block storage in combination with DIR Storage do the trick. Or do u need the speed gain from lvm?

 

[Glowsome]

Dear @Glowsome,
why do you implement it so complicated? Would not the use of OCFS2 on the Block storage in combination with DIR Storage do the trick. Or do u need the speed gain from lvm?

This whole thing started as a project to get around limitations i encountered.
and as a whole, the usage of OCFS2 is oracle, and after the took over MySQL, i dread being locked in by using this.

Due to the nature it grew.. and grew ... and maybe yes its turned into a beast .. but still all in all i'm not after something like a speed-gain.
Concider it a proof of concept, and its still surviving over and over again fulfilling my needs since i stepped in using PVE 5.x
Latest is i migrated all 4 nodes to PVE 8.x.

- Glowsome

 

[Glowsome]

Hello,

About the "sleep 10" added int the unitfile "lvmlockd.service" you do not have to modify the file.

You can create a directory "/etc/systemd/lvmlockd.service.d/" and create a file "sleep.conf" (exemple) with the content

[Service]
ExecStartPre=/usr/bin/sleep 10

or you can create a directory "/etc/systemd/dlm.service.d/" and create a file "sleep.conf"

[Service]
ExecStartPost=/usr/bin/sleep 10

An update can overwrite the file "lvmlockd.service" or "dlm.service" no problem.

Best regards.

Francis

Hi, might be a late reply on this, but if i read correctly then dropping an addition in:

/etc/systemd/lvmlockd.service.d/

This means you are not adding, but actually overriding.
(for reference) see :
https://access.redhat.com/documenta...tion_assembly_working-with-systemd-unit-files

So the correct place to drop these additions (IMHO) would be in:

/etc/systemd/system/servicename.service.d/*

Where in my case, the additions like the sleep timings, aswell as the additional dependancies introduced like After=
Should be dropped there in contrary of your own suggested context.

We want to add conditions or , not override existing ones.

 

[FrancisS]

Hello Glowsome

Yes sorry the correct path for the file is "/etc/systemd/system/lvmlockd.service.d/".

Where in my case, the additions like the sleep timings, aswell as the additional dependancies introduced like After=
Should be dropped there in contrary of your own suggested context.

I do not understand.

Best regards.

Francis