cluster not ready - no quorum? (500) - pvecm expected 1 does nothing

[eru2]

hi folks,
i had a 2 node cluster with a qdevice on a raspberry pi. The Node where i created the cluster failed and i cant get the second cluster to work when i try to start a vm or lxc container i get the error message in the title.

Any hints?

 

[Maximiliano]

Hello,

For a three node cluster (its the same as two node + qdevice cluster, from Corosync's perspective) to work you need to have two out of three votes, i.e. at most one node can be offline and the remaining nodes are able to talk to each other. Nodes without quorum will have their `/etc/pve` in read-only mode and you won't be able to start/stop/create VMs or change configs.

From what you describe, the remaining node and the qdevice were not able to communicate or the qdevice was offline. Could you please send us the output of

pvecm status

from both nodes in the cluster (or one if the other one is still non-operational). And share with us the contents of `/etc/pve/corosync.conf`?

 

[eru2]

This is output of : pvecm status

root@pve-host2:~# pvecm status
Cluster information
-------------------
Name:             Proxmox-Cluster
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Nov 10 17:28:28 2023
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000002
Ring ID:          2.117
Quorate:          No

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      1
Quorum:           2 Activity blocked
Flags:            Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000002          1   A,NV,NMW 192.168.0.9 (local)
0x00000000          0            Qdevice (votes 1)

the second node is broken.

contents of /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: Proxmox
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.0.8
  }
  node {
    name: pve-host2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.0.9
  }
}

quorum {
  device {
    model: net
    net {
      algorithm: ffsplit
      host: 192.168.0.23
      tls: on
    }
    votes: 1
  }
  provider: corosync_votequorum
}

totem {
  cluster_name: Proxmox-Cluster
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

 

[Maximiliano]

Have you restarted your machine? `pvecm status` and the config are not sync and this might happen if, for example, you manually edit the config files but don't restart the Corosync service so it can reload the config. The config has 2 nodes + qdevice in it, while `pvecm status` only shows 1 node + qdevice.

Can you also check if the contents of `/etc/pve/corosync.conf` and `/etc/corosync/corosync.conf` coincide?

 

[eru2]

I got it to work again, with the manual in the Cluster manager wiki
under the point

https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node​