[SOLVED] What's the point of a Separated cluster network?

[aasami]

Hello. I've set up a cluster with physically separated cluster network but it is not doing what I expect it to do.
Here is my lab:

[node1 ~]# pvecm status
Cluster information
-------------------
Name:             clustest
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Jun 19 11:08:54 2020
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1.8b
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 172.16.0.1 (local)
0x00000002          1 172.16.0.2
0x00000003          1 172.16.0.3
[node1 ~]#
[node1 ~]# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: one
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 172.16.0.1
    ring1_addr: 10.1.0.1
  }
  node {
    name: two
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 172.16.0.2
    ring1_addr: 10.1.0.2
  }
  node {
    name: three
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 172.16.0.3
    ring1_addr: 10.1.0.3
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: clustest
  config_version: 3
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

[node1 ~]#
[node1 ~]# cat /etc/pve/datacenter.cfg
console: vv
keyboard: en-us
migration: network=10.1.0.0/16
[node1 ~]#

I have simulated malfunction of the "primary network" (10.1.0.0/16) on node1 so only cluster network (172.16.0.0/24) remained working.
But, to my surprise, no resources have been moved to other nodes.
How should I configure a HA cluster to move resources to other nodes on such malfunction via a cluster network.
Because in this situation the cluster thinks everything is OK (it can communicate through cluster network) and doesn't even try to migrate anything.
Preferably with bandwidth limitation, to not disrupt cluster communication in cluster network?
Something like this in datacenter.cfg?:

fallback_migration: network=172.16.0.0/24
fallback_bwlimit: migration=100000

Please advise.

 

[aasami]

How to migrate VM from such (offline) physical server node to another when only available connection is the separated cluster network? Is it possible?

 

[aaron]

The cluster network for Corosync is used only for that and should only be used for Corosync. More on that later.

Corosync is used to decide on things in the cluster. Things like changing the config of a VM for example.

HA kicks in once a node with HA resources on it does not respond anymore via Corosync either because it is down or cannot talk to the other nodes via Corosync anymore in a timely manner.

An outgoing network interface or migration network interface being down has no effect on Corosync and thus the nodes can still talk to each other via corosync.

To make other networks fault tolerant it is best to use some kind of bonding.

Now, corosync should be run on its own physical network for one reason -> it really needs low latency. Once you start to run other services on the same physical network, these services might congest the network (storage, backups, migration, ...) and thus the latency of the corosync packets can increase to the point where corosync decides that it hasn't heard from the other nodes in time.

The worst case scenario ist that you have HA guests on each node and corosync shares the same physical network with other services that cause the latency to go up. The nodes are not able to communicate with each other in time and the whole cluster falls apart, there is no subsegment of nodes that could still form a quorum.

Once a node with HA enabled cannot talk to the quorum part of the cluster, it will fence itself after ~2 minutes (reset itself). Thus it can appear that your whole cluster restarted itself out of the blue.

To avoid this, the first step is to put corosync on its own physical network. Additionally it is a good idea to define more links for corosync. It can handle a faulty network itself and switch to other links. There is no need to set up a bonded interface for corosync as bonds are usually way slower to switch between physical links than corosync would switch between configured links itself.


I hope this helps a bit to clear up the misunderstanding

 

[aasami]

The cluster network for Corosync is used only for that and should only be used for Corosync.
HA kicks in once a node with HA resources on it does not respond anymore via Corosync.
To make other networks fault tolerant it is best to use some kind of bonding.
I hope this helps a bit to clear up the misunderstanding

Sure it did! The solution to my question was just to shutdown the "offline" node (either physically or via ILO) and it automatically moves all VMs to other nodes according to ha-manager policy. When the (simulated) network issue has been resolved, and the node has been powered back up, VMs returned to their place as if nothing ever happened.
That's what I was after. God bless you my friend and thank you.