OpenVZ migration : Migration Error

[Chris Rivera]

When migrating from an openvz node to another openvz node i get a migration error.

Sep 13 08:35:30 starting migration of CT 334 to node 'proxmox9' (*.217.*.153)
Sep 13 08:35:30 container is running - using online migration
Sep 13 08:35:30 starting rsync phase 1
Sep 13 08:35:30 # /usr/bin/rsync -aHAX --delete --numeric-ids --sparse /var/lib/vz/private/334 root@*.217.*.153:/var/lib/vz/private
Sep 13 08:36:40 start live migration - suspending container
Sep 13 08:36:40 dump container state
Sep 13 08:36:42 copy dump file to target node
Sep 13 08:36:47 starting rsync (2nd pass)
Sep 13 08:36:47 # /usr/bin/rsync -aHAX --delete --numeric-ids /var/lib/vz/private/334 root@*.217.*.153:/var/lib/vz/private
Sep 13 08:36:48 dump 2nd level quota
Sep 13 08:36:48 copy 2nd level quota to target node
Sep 13 08:36:50 initialize container on remote node 'proxmox9'
Sep 13 08:36:50 initializing remote quota
Sep 13 08:36:50 turn on remote quota
Sep 13 08:36:50 load 2nd level quota
Sep 13 08:36:50 starting container on remote node 'proxmox9'
Sep 13 08:36:50 restore container state
Sep 13 08:38:45 # /usr/bin/ssh -c blowfish -o 'BatchMode=yes' root@*.217.*.153 vzctl restore 334 --undump --dumpfile /var/lib/vz/dump/dump.334 --skip_arpdetect
Sep 13 08:36:51 Restoring container ...
Sep 13 08:36:52 Starting container ...
Sep 13 08:36:52 Warning: Unknown iptable module: xt_connlimit, skipped
Sep 13 08:36:52 Container is mounted
Sep 13 08:36:52 undump...
Sep 13 08:36:52 Setting CPU units: 1000
Sep 13 08:36:52 Setting CPUs: 1
Sep 13 08:36:52 Configure veth devices: veth334.0
Sep 13 08:36:52 Adding interface veth334.0 to bridge vmbr0 on CT0 for CT334
Sep 13 08:38:45 vzquota : (warning) Quota is running for id 334 already
Sep 13 08:38:45 ERROR: online migrate failure - Failed to restore container: interrupted by signal
Sep 13 08:38:45 removing container files on local node
Sep 13 08:38:47 start final cleanup
Sep 13 08:38:48 ERROR: migration finished with problems (duration 00:03:18)
TASK ERROR: migration problems

I believe this is due to the new openvz node not having the same iptables modules loaded from the original openvz node.


When the migration is complete and shows online on the new node and i get migration error. The vm is locked and not in a functional state. I cannot shutdown, console, or stop the vm.

Warning: Unknown iptable module: xt_connlimit, skipped
Container already locked
TASK ERROR: command 'vzctl stop 334 --fast' failed: exit code 9


I need to manually goto the openvz lock folder and remove the locked file. Then have to find the pids and kill them to reboot the vm.

I have made notes of this because this is a constant issue i have when migrating openvz containers

# get list of pid and child pids
pstree -nup | grep init


# match pid to get ctid
vzpid {ID} // to get ct idnum


# once you find the right pid for the ct
# kill the process and child processes using kill {PID}
kill {PID}



The reason why i cannot just fix the iptables issue is because i need to restart openvz for it to take into effect the updated modules which will cause downtime for customers. Downtime needs to be scheduled for this to happen since high availability is not configured.

This takes a lot of manual intervention. Is it possible for migration to detect these settings before the migration process begins? If it did then it would not migrate a vm to a node that is not capable and would not require someone to login via console to correct such issues. this will ensure that during migration the vm will not encounter issues when cleaning up, maximizing uptime for end users=>clients

Sep 13 08:36:51 Restoring container ...
Sep 13 08:36:52 Starting container ...
Sep 13 08:36:52 Warning: Unknown iptable module: xt_connlimit, skipped
Sep 13 08:36:52 Container is mounted
Sep 13 08:36:52 undump...
Sep 13 08:36:52 Setting CPU units: 1000
Sep 13 08:36:52 Setting CPUs: 1
Sep 13 08:36:52 Configure veth devices: veth334.0
Sep 13 08:36:52 Adding interface veth334.0 to bridge vmbr0 on CT0 for CT334
Sep 13 08:38:45 vzquota : (warning) Quota is running for id 334 already
Sep 13 08:38:45 ERROR: online migrate failure - Failed to restore container: interrupted by signal
Sep 13 08:38:45 removing container files on local node
Sep 13 08:38:47 start final cleanup
Sep 13 08:38:48 ERROR: migration finished with problems (duration 00:03:18)
TASK ERROR: migration problems

 

[dietmar]

I guess Migration a container which use iptables is unsafe anyways. Bu not sure how we should detect that?

 

[Chris Rivera]

When your offering hosting to clients your job is to provide them with what they want... or they'll go somewhere else. Another way around this would be to provision a kvm but users dont tell you their iptables specifications when ordering.

In the openvz.conf file : This allows your to specify what iptables modules can be loaded by the openvz containers.

http://blog.bodhizazen.net/linux/proxmox-using-iptables-in-openvz-guests/


We have clients who need certain modules to run VOIP and other types of systems so if they are on a node with certain modules loaded will cause a failed migration when migrating to another node it would be nice for the admin to know this before causing the end user => client any downtime.

Also if the script can detect that something went wrong with the migration but took the steps that i mentioned:

delete the lock file
kill the process
start the vm again

That would also work. This decrease the downtime that a vm / client would experience and increase their up time (Our Mission).