Migrate VM from offline node

[Erik_ch]

I run a cluster with three nodes (quorum is given) normally. The VM's are synchronized with DRBD or on NAS share (uncritical VMs). HA groups aren't used at the moment (I'm still in trial and learning status with this functionality.)

Yesterday I had a crash on node2 caused by a driver. I was clear that I needed some time to restore it. I thought that I should be able to migrate (offline) my crashed VM's to node1 or node3. But fail: PVE told me that there is no connection to node2! This is clear but I know that path "/etc/pve" is available on all nodes synchronized.

As consequence I logged in the console of node3 and I moved vm's conf files from node2 to node3 manually. Afterwards I was able to start VM's on node 3. Even after restored node2 there were no issue because node2 got the latest pve configs from corosync and manually "migrated" VMs didn't start there correctly.

Why isn't PVE 4.3 not able to "migrate" or change VMs from "dead" nodes to another "living" node? The only condition is a non-local storage.

 

[tom]

This works automatically if you use HA.

See http://pve.proxmox.com/wiki/High_Availability_Cluster_4.x

If you do not use HA, you have to move the VM config manually.

 

[Erik_ch]

Thanks for your answer. I know about HA 4 functionality but I hesitate to use it caused by some disadvantages.

The manual transfer from a "dead" server to a running server within a cluster is secure, because I know if a server is really down or not. So this "half" HA allows running one server in emergency case and take over all VMs from crashed cluster. Such a special offline migration helps to save time.

I tested your HA functionality with simulator and within a test environment today. It's a very good solution for if one server crashes by a hardware failure. But the main disadvantage is that you have to have two servers in minimum.

In case of power loss I use an usv battery. After some minutes I want to start a script to migrate critical VMs to one server and shutdown all others to save power. So I can extend the duration of usv from some 15 minutes to more than one hour. But HA kills then all VMs because the quorum has been lost instead of just let last server independent of it's state. It's OK if pvecm sets /etc/pve to read only.

I'm thinking about a fourth small quorum server based on APU2 or PI3 which has no further functionality. Such a "server" consumes some watts only.

 

[fabian]

Thanks for your answer. I know about HA 4 functionality but I hesitate to use it caused by some disadvantages.

The manual transfer from a "dead" server to a running server within a cluster is secure, because I know if a server is really down or not. So this "half" HA allows running one server in emergency case and take over all VMs from crashed cluster. Such a special offline migration helps to save time.

I tested your HA functionality with simulator and within a test environment today. It's a very good solution for if one server crashes by a hardware failure. But the main disadvantage is that you have to have two servers in minimum.

In case of power loss I use an usv battery. After some minutes I want to start a script to migrate critical VMs to one server and shutdown all others to save power. So I can extend the duration of usv from some 15 minutes to more than one hour. But HA kills then all VMs because the quorum has been lost instead of just let last server independent of it's state. It's OK if pvecm sets /etc/pve to read only.

I'm thinking about a fourth small quorum server based on APU2 or PI3 which has no further functionality. Such a "server" consumes some watts only.

that setup sounds like it is not a good idea for "critical VMs". UPS like that are for keeping servers running long enough for an orderly shutdown. if you want more, then you need either a very beefy UPS or a UPS with a generator, and then you probably want to keep your cluster running regularly and not introduce extra load and instability by migrating VMs and shutting down half your cluster.

 

[Erik_ch]

For most cases / vms you are right. Orderly shutdown is necessary for all servers at the end. I only want to let run the asterisk based PBX (+switch/+router) as long UPC reaches 10% of power (ca. 1 hour).

Running "half" cluster combined with manual recovery (important) isn't so risky. In my scenario the flag "nofailback" must be set. Before migrating back after reboot, DRBD status has to be checked and restored manually before migration back is possible. If there were a split brain I know which server has to survive.

Anyway thanks your answers. I will rethink about my concept and test it.

 

[LnxBil]

For most cases / vms you are right. Orderly shutdown is necessary for all servers at the end. I only want to let run the asterisk based PBX (+switch/+router) as long UPC reaches 10% of power (ca. 1 hour).

For what? Is all your infrastructure like switches, media converters, even modems and your telephones with maybe office switches in between on UPS aswell? Nice if everything works except no one can use it. It's always very interesting to look at the faces of managers if you calculate what that kind of HA costs - you always need a generator or a new cable management for UPS powered emergency lines.

Running "half" cluster combined with manual recovery (important) isn't so risky. In my scenario the flag "nofailback" must be set. Before migrating back after reboot, DRBD status has to be checked and restored manually before migration back is possible. If there were a split brain I know which server has to survive.

Anyway thanks your answers. I will rethink about my concept and test it.

Also calculate the ROI value. A simple generator can be cheaper than testing all the possibilities of your manual-ha-failover-still-something-on concept that has to be done to work reliable. With DRBD it is always very tricky because you only need one situation which screws with your data and you've a split brain. Believe me, I've been there and learned a lot and try to avoid the possibility to happen in the first place.

 

[Erik_ch]

LnxBil Thanks for your tip.
For what? For PBX (Asterisk), some phone (via PoE), Switche and router. I need to run PBX infrastructure for about 60 min. after power loss. There are two reasons: a) evacuation announcement to phones via multicast and b) emergency call from elevators to support center.

So I have two scenarios to handle:
a) Power loss > 5min.: Hold a minimum infrastructure for 60 min. (see above) and shutdown (proper) what isn't needed to save power. In this worse case scenario real proper HA isn't so important. A recovery needs some manual intervention.
b) Server crash, server maintenance or power loss < 5 min.: In this case I want to have a proper HA with quorum (proxmox and DBRD). In this case DRBD status has to be checked before migrate back to crashed server and in case of split brain last shutdowned server must bootstrap DRBD. "Nofailback" ist standard for me because DRBD needs some time for resync.

A generator isn't a valuable option for our power need. Self starting generator are expensive and needs a lot of maintenance. My 3kVA usv holds about 80 min. with minimum of infrastructure. Under full load it's about 15 min.

With DBRD I have had experience for about 4 years. I learned a lot too. I know how critical split brain is. If UPS software agent shutdowns uncritical vms and servers properly in scenario 1 split brain is no issue. I know which server stopped as last.

Anyway I don't want to bother you. It was an interesting discussion. Thank you all.