Unable to move LXC to or from one host in the cluster

[pmoxhome99]

Running into an issue in my cluster where I'm not able to move LXC's to and from one of the hosts.

Running a 3 host cluster on 8.2.5, with shared ceph storage.

When I initiate an LXC migration it starts the vzmigrate task then nothing else happens, no LXC shutdown nothing.
LXC migrations between the other two nodes are working fine

This just started happening, after a memory upgrade of the affected host, so maybe that's related but I'm hoping to be able to move the LXC's off the host somehow before I shut it down/reboot it and investigate the memory upgrade further.

Any idea what might be causing it?

 

[gfngfn256]

but I'm hoping to be able to move the LXC's off the host somehow before I shut it down/reboot it and investigate the memory upgrade further.

Can you backup that LXC to shared storage (maybe that ceph pool?) & then restore it on a different node.

 

[pmoxhome99]

left a few migration jobs running over night and eventually they completed, the were stuck at vzmigrate for a good 5 hours and then eventually the rest of the process kicked in.

I'll start migration jobs for the rest and see where it goes.

 

[alexskysilk]

Any idea what might be causing it?

containers are cold migrated only, which means they need to be shut down prior to being migrated. If the container refuses to shut down (which happens) they process will appear to be stuck.

the remedy is to find and kill the lxc-start process for that container, and the migration will resume.

 

[pmoxhome99]

aware of that and I had manually killed several of the LXC's before starting migration and even those failed to migrate.

 

[alexskysilk]

even those failed to migrate.

then that is a different but related problem- its an RBD that's stuck with open files (which also explains why container wouldnt shut down.) I have a script for these eventualities.

#!/bin/bash
# usage: rbdrelease ctid
if [ -z $1 ]
then
        echo ctid not provided!
        exit
fi
ctid=$1

rbdpath=$(find /dev/rbd -name "vm-$1*")
if [ -z $rbdpath ]
then
        echo "ct $ctid not mounted on this node!"
        exit
fi
pool=$(echo $rbdpath | cut -d/ -f4)
rbdid=/dev/$(ls -la /dev/rbd/"$pool" | grep "$ctid" | cut -d/ -f3)
procids=$(fuser -am $rbdid | cut -d':' -f2 | tr -d '[:blank:]')
if [ -z "$procids" ]; then
        rbd unmap -o force "$rbdid"
        echo "rbd for ct $ctid released"
else echo "$rbdid for ct $ctid is still in use!"
fi