HA-migrate when bringing node down fails

[mir]

Hi,

Doing a HA-migrate manually completes as expected but when the migration needs to be done automatically because the node is manually requested to shutdown it fails.

The log for task shows the following:
The shutdown task as a consequence of "Stop all VMs and Containers"
task started by HA resource agent
Stopping container ...
Container was stopped
Container is unmounted
TASK OK

Next task should then start the VM on a new node:
task started by HA resource agent
Starting container ...
Container private area /var/lib/vz/private/103 does not exist
TASK ERROR: command 'vzctl start 103' failed: exit code 43

The cause for the failure is that the Container private area is never transferred to the other node. Shouldn't the container private area be transferred before shutting down?


Log from a manual migration:
200 OK
[
{
"n" : 1,
"t" : "task started by HA resource agent"
},
{
"n" : 2,
"t" : "Jul 30 04:02:50 starting migration of CT 103 to node 'esx1' (192.168.2.8)"
},
{
"n" : 3,
"t" : "Jul 30 04:02:50 container is running - using online migration"
},
{
"n" : 4,
"t" : "Jul 30 04:02:50 starting rsync phase 1"
},
{
"n" : 5,
"t" : "Jul 30 04:02:50 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/103 root@192.168.2.8:/var/lib/vz/private"
},
{
"n" : 6,
"t" : "Jul 30 04:05:25 start live migration - suspending container"
},
{
"n" : 7,
"t" : "Jul 30 04:05:25 dump container state"
},
{
"n" : 8,
"t" : "Jul 30 04:05:26 copy dump file to target node"
},
{
"n" : 9,
"t" : "Jul 30 04:05:26 starting rsync (2nd pass)"
},
{
"n" : 10,
"t" : "Jul 30 04:05:26 # /usr/bin/rsync -aH --delete --numeric-ids /var/lib/vz/private/103 root@192.168.2.8:/var/lib/vz/private"
},
{
"n" : 11,
"t" : "Jul 30 04:05:26 dump 2nd level quota"
},
{
"n" : 12,
"t" : "Jul 30 04:05:26 copy 2nd level quota to target node"
},
{
"n" : 13,
"t" : "Jul 30 04:05:28 initialize container on remote node 'esx1'"
},
{
"n" : 14,
"t" : "Jul 30 04:05:28 initializing remote quota"
},
{
"n" : 15,
"t" : "Jul 30 04:05:28 turn on remote quota"
},
{
"n" : 16,
"t" : "Jul 30 04:05:28 load 2nd level quota"
},
{
"n" : 17,
"t" : "Jul 30 04:05:28 starting container on remote node 'esx1'"
},
{
"n" : 18,
"t" : "Jul 30 04:05:28 restore container state"
},
{
"n" : 19,
"t" : "Jul 30 04:05:28 removing container files on local node"
},
{
"n" : 20,
"t" : "Jul 30 04:05:38 start final cleanup"
},
{
"n" : 21,
"t" : "Jul 30 04:05:42 migration finished successfuly (duration 00:02:53)"
},
{
"n" : 22,
"t" : "TASK OK"
}
]

 

[dietmar]

Container storage needs to be on shared storage. Else HA will not work.

See http://pve.proxmox.com/wiki/High_Availability_Cluster#System_requirements

 

[mir]

Container storage needs to be on shared storage. Else HA will not work.

See http://pve.proxmox.com/wiki/High_Availability_Cluster#System_requirements

Ah, I see. So:
1) CTs can be live migrated manually
2) CT's cannot be migrated at all automatically by HA

Is the above assumptions correct?

 

[dietmar]

1) CTs can be live migrated manually
2) CT's cannot be migrated at all automatically by HA

Is the above assumptions correct?

No, you can migrate as long as the HA stack thinks the VM is online.

Note: running a CT with HA without some kind of shared/replicated storage makes no sense at all.

 

[mir]

No, you can migrate as long as the HA stack thinks the VM is online.

Can you elaborate some more? Or is this the reason for shared storage? Eg the HA agent is only capable of stopping and starting a VM given the place for the private area is "fixed"?

Note: running a CT with HA without some kind of shared/replicated storage makes no sense at all.

I agree I am only testing the corners of PVE

 

[dietmar]

I agree I am only testing the corners of PVE

Or you probably do not understand the issue.

 

[mir]

Or you probably do not understand the issue.

It was triggered by your comment that live migration was possible with CTs as a reply to my comment that CTs could not be used for HA as long as OpenVZ does not support shared storage (I don't see NFS as an alternative here).

 

[tom]

if you store the data locally and you lost this node, how should HA start the CT on another node without the data?

therefore you need store the data on NFS. clear now?