Extra container mount point and HA migration

[Max Timokhin]

I have two nodes cluster setup (actually three nodes with the third used for quorum). I'm using LVM volume storage (not thin provisioning) on top of DRBD on first two nodes. The LVM storage is marked as "shared". I created an LXC container and can migrate it between nodes without any problem, including HA scenario.

Now I need to connect another storage disk (from the LVM-over-DRBD storage) to that container. For testing purposes I added

mp0: vg0:vm-100-disk-2,mp=/mnt,size=512M

to configuration of the container.

The container mounts the volume successfully. But now I can not migrate container to another node: Proxmox can not deactivate second LVM volume because there is mounted filesystem in use.

So, what are my options to automatically migrate an LXC container with extra mount point? I will try to mount the second volume from the container itself (with all the extra LXC/AppArmour settings), are there any other ways? Any difference if using KVM instead of LXC?

--
Best regards, Max.

 

[Max Timokhin]

The migration log is:

task started by HA resource agent
2018-01-29 21:21:33 starting migration of CT 100 to node 'n1' (192.168.88.137)
2018-01-29 21:21:33 volume 'vg0:vm-100-disk-1' is on shared storage 'vg0'
2018-01-29 21:21:33 volume 'vg0:vm-100-disk-2' is on shared storage 'vg0'
can't deactivate LV '/dev/vg0/vm-100-disk-2': Logical volume vg0/vm-100-disk-2 contains a filesystem in use.
2018-01-29 21:21:33 ERROR: volume deactivation failed: vg0:vm-100-disk-2 at /usr/share/perl5/PVE/Storage.pm line 999.
2018-01-29 21:21:33 aborting phase 1 - cleanup resources
2018-01-29 21:21:33 start final cleanup
vm 100 - lxc.aa_profile is deprecated and was renamed to lxc.apparmor.profile
2018-01-29 21:21:33 ERROR: migration aborted (duration 00:00:01): volume deactivation failed: vg0:vm-100-disk-2 at /usr/share/perl5/PVE/Storage.pm line 999.
TASK ERROR: migration aborted

The same result even for offline container (stop the container manually, trying to migrate, see the log above). lsof shows nothing on both nodes.

--
Best regards, Max.

 

[wolfgang]

Hi,

please try the following command when the container is offline.

/sbin/vgchange -aln  vg0/vm-100-disk-2

 

[Max Timokhin]

Hello, Wolfgang.

I've traced an issue via Proxmox sources and yes, this is the command that caused the error. The command prints the same error message (filesystem is in use). The Proxmox code for mounting/unmounting/migration looks correct (and it's rather easy to read btw).

lvs shows attribute o for vm-100-disk-2 even in offline mode, so formally the volume is mounted. The device presents at sys/fs/ext4, but holders subdirectory is empty. lsof returns empty results. Looks like some LXC bug maybe, have absolutely no idea (.

 

[wolfgang]

Looks like some LXC bug maybe, have absolutely no idea

I tried it with latest PVE on a plain local LVM and everything worked fine.
You should search at DRBD maybe something wrong here?

 

[Max Timokhin]

Wolfgang, thanks for testing.

DRBD device is a physical volume here, so it's more lower level I believe, and the main logical volume (disk-1) is on the same VG and the same DRBD device. My PVE version is 5.1 with latest updates. I'll try some another tests and write about results. Just some questions:

1. Is it correct (by PVE design) to use multiple disks for HA container instance?
2. Should I use shared=1 for mp0 declaration (tryed with and without it actually)?
3. Is it a good idea to mount a volume using device mount point and container's own mount/fstab?

 

[wolfgang]

Now I understand what the problem is.
You use a single LVM and this can only be active on one Node. There is no active active mode in DRBD for long usage.
You have to use the DRBD Storage Plugin, this creates for every volume a independent LV with DRBD.
It is available from Linbit.

 

[Max Timokhin]

Hmmm, so https://pve.proxmox.com/wiki/DRBD is (completely) obsolete and "shared" LVM storage works only for "real" shared storage. But if a container has only one LVM volume, works anyway .

Ok, many thanks, I'll try DRBD9/drbdmanage/storage plugin.

--
Best regards, Max

 

[wolfgang]

The wiki is only valid for PVE3.x and DRBD 8 as the node on the top suggest.

We do not support DRBD anymore.
So I can't tell you what is working and we do not check the DRBD Plugin.