cluster HA not moving configuration for CT with bind mount

[xav]

Hello,

I have some CT using some bind mount using a VMID.mount file.
When failing over to an other node using cluster HA. Those CT are not started on the failover node.

Is there an issue with cluster HA and bind mounts ?

Seems that .mount scripts are not dispatched across cluster nodes

Regards,

xav

 

[dietmar]

Seems that .mount scripts are not dispatched across cluster nodes

All scripts are on the distributed file system, so I am quite sure they are there (but that is easy to check).

Any info in the logs?

 

[xav]

Hello,

here is the whole process:

CT 104 is running on srv4

srv4:~# vzlist 
CTID      NPROC STATUS    IP_ADDR         HOSTNAME
104         10 running   192.168.0.44    ha104.domain.net

104.mount file exists on node srv4

srv4:~# ls /etc/pve/nodes/srv4/openvz/104.mount -all
-rwxr----- 1 root www-data 467 Apr 11 04:57 /etc/pve/nodes/srv4/openvz/104.mount

104.mount file content on node srv4

srv4:~# cat /etc/pve/nodes/srv4/openvz/104.mount
#!/bin/bash
source /etc/vz/vz.conf
source ${VE_CONFFILE}
mount --bind /home/nas/portage ${VE_ROOT}/usr/portage

zpool-001227 is well mounted on node srv4

srv4:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
none                  12329000       260  12328740   1% /dev
/dev/md1              20317384   2567852  16725588  14% /
tmpfs                 12338816         0  12338816   0% /lib/init/rw
tmpfs                 12338816     44572  12294244   1% /dev/shm
/dev/mapper/pve-data 1916756132   2457412 1817699888   1% /var/lib/vz
10.16.100.254:/zpool-001227/data
                     262144000  11790336 250353664   5% /home/nas

Bind mount is well mounted on CT 104 while running on srv4

srv4:~# vzctl enter 104
entered into CT 104
ha104 / # df
Filesystem                       1K-blocks     Used Available Use% Mounted on
/dev/simfs                        52428800  4164608  48264192   8% /
10.16.100.254:/zpool-001227/data 262144000 11790336 250353664   5% /usr/portage
cgroup_root                          10240        0     10240   0% /sys/fs/cgroup
udev                                 10240        4     10236   1% /dev
none                               2097152        0   2097152   0% /dev/shm

srv4 is the master, srv5 has no container running

srv5:~# vzlist 
Container(s) not found

104.mount doesn't exists on srv5 (node srv4 is master)

srv5:~# ls /etc/pve/nodes/srv5/openvz/104.mount 
ls: cannot access /etc/pve/nodes/srv5/openvz/104.mount: No such file or directory

we reboot srv4

srv4:~# reboot

srv5 is taking over HA containers, and starts 104

==> /var/log/cluster/rgmanager.log <==
Apr 11 10:41:18 rgmanager Member 1 shutting down
Apr 11 10:41:19 rgmanager Starting stopped service pvevm:104
Apr 11 10:41:19 rgmanager [pvevm] Move config for CT 104 to local node
Apr 11 10:41:20 rgmanager [pvevm] Task still active, waiting
Apr 11 10:41:21 rgmanager Service pvevm:104 started

CT 104 is running on srv5

srv5:~# vzlist 
CTID      NPROC STATUS    IP_ADDR         HOSTNAME
104         10 running   192.168.0.44    ha104.domain.net

there is no 104.mount on srv5

srv5:~# ls /etc/pve/nodes/srv5/openvz/104.mount 
ls: cannot access /etc/pve/nodes/srv5/openvz/104.mount: No such file or directory

zpool-001227 is well mounted on node srv5

srv5:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
none                  12329000       260  12328740   1% /dev
/dev/md1              20317384   1055040  18238400   6% /
tmpfs                 12338816         0  12338816   0% /lib/init/rw
tmpfs                 12338816     38308  12300508   1% /dev/shm
/dev/mapper/pve-data 1916756132    456792 1819700508   1% /var/lib/vz
10.16.100.254:/zpool-001227/data
                     262144000  11790336 250353664   5% /home/nas

we enter CT 104 on srv5 and see that bind mounts are not mounted

srv5:~# vzctl enter 104
entered into CT 104
ha104 / # df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/simfs      52428800 4163584  48265216   8% /
cgroup_root        10240       0     10240   0% /sys/fs/cgroup
udev               10240       4     10236   1% /dev
none             2097152       0   2097152   0% /dev/shm

we reboot srv5

srv5:~# reboot

srv4 is taking over

Apr 11 10:44:00 rgmanager Migration: pvevm:104 is running on 2
Apr 11 10:45:03 rgmanager Member 2 shutting down
Apr 11 10:45:03 rgmanager Starting stopped service pvevm:104
Apr 11 10:45:04 rgmanager [pvevm] Move config for CT 104 to local node
Apr 11 10:45:05 rgmanager [pvevm] Task still active, waiting
Apr 11 10:45:06 rgmanager Service pvevm:104 started

CT 104 is now running on srv4

srv4:~# vzlist 
CTID      NPROC STATUS    IP_ADDR         HOSTNAME
104         10 running   192.168.0.44    ha104.domain.net

bind mounts are mounted as expected

srv4:~# vzctl enter 104
entered into CT 104
ha104 / # df
Filesystem                       1K-blocks     Used Available Use% Mounted on
/dev/simfs                        52428800  4164608  48264192   8% /
10.16.100.254:/zpool-001227/data 262144000 11790336 250353664   5% /usr/portage
cgroup_root                          10240        0     10240   0% /sys/fs/cgroup
udev                                 10240        4     10236   1% /dev
none                               2097152        0   2097152   0% /dev/shm

So, something doen't go well with bind mounts and 104.mount is not moved around while failover.
am I doing something wrong ?

Do you need specific tests ?

regards,

xav

 

[tom]

I also see this, pls file a bug at https://bugzilla.proxmox.com

 

[xav]

I also see this, pls file a bug at https://bugzilla.proxmox.com

Done https://bugzilla.proxmox.com/show_bug.cgi?id=158

xav

 

[xav]

Work around at the moment is to copy the VMID.mount file to all cluster nodes.

xav

 

[xav]

Bug is fixed https://bugzilla.proxmox.com/show_bug.cgi?id=158
Can't wait to update !

Thanks

xav

 

[xav]

Working great with pve-manager: 2.0-59
thanks

xav