pvecm expected causing problems

[freebee]

I have 5 servers in a cluster. Previously, to maintain the servers, I configured pvecm expected 1 and migrated the vms, thus being able to update the servers individually. Now, I can't repeat this process because when I restart a server the others automatically restart bringing down the entire cluster. I saw in other topics that it is due to a change in corosync, but the question remains: does this change make sense?. Because if the cluster is about availability, it does not make sense to impose this because it ends up taking down all services in a simple restart of just one server.

pve-manager/6.4-6/be2fa32c (running kernel: 5.4.114-1-pve)

pvecm status
Cluster information
-------------------
Name: Cluster0
Config Version: 5
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Wed Jul 7 12:06:14 2021
Quorum provider: corosync_votequorum
Nodes: 5
Node ID: 0x00000001
Ring ID: 1.1b0c
Quorate: Yes

Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 5
Quorum: 3
Flags: Quorate


I try to set to 3:

pvecm expected 3
Unable to set expected votes: CS_ERR_INVALID_PARAM.

 

[t.lamprecht]

I have 5 servers in a cluster. Previously, to maintain the servers, I configured pvecm expected 1 and migrated the vms, thus being able to update the servers individually.

Setting expected votes should only be done when fixing things gone wrong, not for normal maintenance.

Now, I can't repeat this process because when I restart a server the others automatically restart bringing down the entire cluster.

That itself does not sound normal either, with a 5 node cluster you should be able to reboot one node just fine without any problems for the others, even two nodes could be possible, as the remaining three ones would still provide more than 50% of the total votes, and so quorum.
And it can work, I know because I use that almost daily in some more permanent three-node testlab cluster I'm doing lots of (ugly) things on.

Because if the cluster is about availability, it does not make sense to impose this because it ends up taking down all services in a simple restart of just one server.

Can you please post the full pveversion -v output? Also what is in the syslog of the other nodes during a reboot of one node?

 

[freebee]

root@HV01:~# pvecm expected 3
Unable to set expected votes: CS_ERR_INVALID_PARAM
root@HV01:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-4
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

The syslog is attached. The problem was around 15:13.

 

[freebee]

Workaround:

With the problem i find a way to disable the reboot of nodes.

In corosync has a option: two_node
Now i can migrate VMs in servers and make the proxmox upgrades without worry about reboot all servers in cluster.
From man:

The "two node cluster" is a use case that requires special consideration. With a standard
two node cluster, each node with a single vote, there are 2 votes in the cluster. Using
the simple majority calculation (50% of the votes + 1) to calculate quorum, the quorum
would be 2. This means that the both nodes would always have to be alive for the cluster
to be quorate and operate.

Enabling two_node: 1, quorum is set artificially to 1.

The procedure:

stop cluster services and corosync on all servers in the cluster.
systemctl stop corosync
systemctl stop pve-cluster

After do this in all servers, in just one server (choose one to reboot after procedure) do:
pmxcfs -l and edit the file /etc/pve/corosync.conf line:

quorum {
provider: corosync_votequorum
}

to:

quorum {
provider: corosync_votequorum
two_node: 1
}

after change, reboot this server.

He will back with pve-cluster and corosync started. So, in the other servers do:
systemctl start corosync
systemctl start pve-cluster

you can verify if the local corosync is updated:
cat /etc/corosync/corosync.conf

He must be updated on line included.

If you want to make some experiences (like just two servers and one reboot), search the wait_for_all option on man. The cluster will not stop.