[SOLVED] Node lost connection to cluster after changing quorum votes?

MightyMauz

Member
Apr 11, 2023
10
1
8
Netherlands
I changed my 3 node cluster setup and in the process I lost connection to 1 of my nodes

The cluster has 3 nodes
- apocalypse (10.12.23.34)
- xavier (10.12.23.31)
- magneto (10.12.23.33)

I wanted to make xavier the main node, as it will always be online. Therefore I changed the votes from 3, 1, 1 to 1, 3, 1 with the help of this video https://www.youtube.com/watch?v=sjS9oDEw9EQ by Apalrd. It instructs to update the corosync.conf by changing the votes and updating the version.

Never had any problems doing this, but today, when all was running again, node magneto did not come along. I can still access it, and get to the webgui, but it has lost connection to the cluster.

How can I fix this?


Here are the corosync.conf files

apocalypse
Code:
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: apocalypse
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.12.23.34
  }
  node {
    name: magneto
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.12.23.33
  }
  node {
    name: xavier
    nodeid: 1
    quorum_votes: 3
    ring0_addr: 10.12.23.31
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: X
  config_version: 15
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

xavier
Code:
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: apocalypse
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.12.23.34
  }
  node {
    name: magneto
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.12.23.33
  }
  node {
    name: xavier
    nodeid: 1
    quorum_votes: 3
    ring0_addr: 10.12.23.31
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: X
  config_version: 15
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

magneto
Code:
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: apocalypse
    nodeid: 3
    quorum_votes: 3
    ring0_addr: 10.12.23.34
  }
  node {
    name: magneto
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.12.23.33
  }
  node {
    name: xavier
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.12.23.31
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: X
  config_version: 14
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}
 
magneto showing v14 while the other two show v15 is where I’d start. On xavier and apocalypse, run pvecm status and check that both see the same two members and report Quorate: Yes. On magneto, check journalctl -b -u corosync and compare its /etc/corosync/corosync.conf with the working /etc/pve/corosync.conf from xavier. The path matters because Proxmox keeps a cluster copy and a local copy. Proxmox configuration docs

If the pair is quorate, Corosync fails to start on magneto, and its local file is still at v14, back up that local file and replace it with the full working v15 file. Keep the healthy pair running. If you use HA, check where the guests are running before restarting Corosync on magneto. Then check pvecm status on xavier again for all three members. Proxmox recovery notes
 
Xavier gets 3 votes, this means your cluster is frozen when it is restarted? All VMs with HA will get frozen too IMHO. If Xavier is broken, then you'll have a permanently dead cluster because you can't do anything anymore. No new nodes, no migration to another cluster l, nothing.
Why do you have a cluster at all if this one node makes it breaks everything?
What do you want to accomplish with this?
 
If the pair is quorate, Corosync fails to start on magneto, and its local file is still at v14, back up that local file and replace it with the full working v15 file. Keep the healthy pair running. If you use HA, check where the guests are running before restarting Corosync on magneto. Then check pvecm status on xavier again for all three members. Proxmox recovery notes
THANKS! That did the trick. After changing /etc/corosync/corosync.conf and a service reboot with systemctl restart corosync the node immediately came back to the cluster.
 
Xavier gets 3 votes, this means your cluster is frozen when it is restarted? All VMs with HA will get frozen too IMHO. If Xavier is broken, then you'll have a permanently dead cluster because you can't do anything anymore. No new nodes, no migration to another cluster l, nothing.
Why do you have a cluster at all if this one node makes it breaks everything?
What do you want to accomplish with this?
  1. I am running a homelab, breaking stuff comes naturally
  2. My networking node (xavier) needs to be able to start, no matter what
  3. I am not using HA and unlikely to do in the near future
  4. I want to manage all nodes from the same webgui
So yeah, a cluster with unequal votes does the trick for me.