[SOLVED] Quorum: 3 Activity blocked

[halley80]

Hi,

My cluster consists of five nodes.
Following a sudden change in the ips vlan of my nodes, I modified the configuration files (network/interface).

For four nodes the 'corosync/pvecm status' is OK.
However, for the last node, I made the command => pvecm expected 1.

Since then, it's not recognized in the cluster but I can access it and the other VMs => cf screenshot

*******************************************
Votequorum information
----------------------
Expected votes: 5
Highest expected: 5
Total votes: 1
Quorum: 3 Activity blocked
Flags:
*******************************************

Do I have to run the node join to the cluster again?
Thanking you for your answers,
Yours sincerely,

 

[halley80]

Now, when i do pvecm staus, i have :

root@ipmpve6:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

Thanks for your help

 

[t.lamprecht]

Following a sudden change in the ips vlan of my nodes, I modified the configuration files (network/interface).

Sudden change? How's that? I guess you did not adapted the corosync configuration(s).

However, for the last node, I made the command => pvecm expected 1.

Why and under what assumption did you do that?? And did you make changes to any configuration files on /etc/pve or /etc/corosync after that?

Please check and post the outputs of:
* on the node with problems systemctl status pve-cluster corosync
* post from the problematic node cat /etc/corosync/corosync.conf
* post from the working node cat /etc/pve/corosync.conf

(note the different paths)

 

[halley80]

Hi,

Thank you for your response.

The change of Vlan is due to a request to a pole of our university.
But it did not go well ....
I didn't change the corosync file
but just the network/interfaces for the Vlan assignment 'Tag VLAN' => Ex: enp94s0f0np0.10

It worked well on the other node.
implies but not this one.
Attach the requested files.
Thanking you,
Yours sincerely,

 

[ermanishchawla]

From the information you shared, servers are not able to communicate to each other, if you have recently changed the vlan membership, please note they must be on same VLAN or reachable for cluster to work
can you please post the
cat /etc/hosts
and output of the following command

ping cluster-ipm1
ping cluster-ipm2
ping cluster-ipm3
ping cluster-ipm4
ping cluster-ipm5
ping cluster-ipm6
ping cluster-ipm7

from one of the server

 

[t.lamprecht]

Additionally to that check if multicast is still allowed and works for that node.

You may then, after you fixed the /etc/hosts (or changed corosync.conf to use IPs not hostnames for the ringX_addr entries):

systemctl reset-failed pve-cluster
systemctl restart corosync pve-cluster

on that node.

 

[halley80]

Hi,

Thank you for your answers.
Attached is the host file and the configuration of the ipm6 node retrieved from node 1.
The hosts file is identical on all nodes and ipm6 ping the other nodes without any problem.

In fact, I am no longer any files in the /etc/pve directory on node 6.

Do you think I can retrieve them from another node, such as ipm1?
Thanking you for your answers,
Yours sincerely,

 

[ermanishchawla]

check corosync configuration, bindnetaddr must be IP not a network address
Please add the correct IP of the node and then restart the following

systemctl restart pve-cluster
systemctl restart corosync

put the output of the following command after the above

pvecm status

 

[halley80]

Thank you for your response but:
the /etc/pve directory is empty ....

and the ping works correctly from this node (6)

 

[halley80]

root@ipmpve6:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

 

[t.lamprecht]

Thank you for your response but:
the /etc/pve directory is empty ....

Your pmxcfs does not starts, kan you check more of the log for the actual reason:
journalctl -b -u pve-cluster

Or try to start it manually: pmxcfs

 

[halley80]

Hi,

Thank you for your response.
I reinstalled node 6 (cleaner).
I backed up my VM(s), turn off the node and run the command => pvecm delnode.

Then I reinstalled it with the same name/ip.
Is it a problem when I'm going to reintegrate it in the cluster ?

Thank you for your answers,
Yours sincerely,

 

[spirit]

but just the network/interfaces for the Vlan assignment 'Tag VLAN' => Ex: enp94s0f0np0.10

interfaces name can't be more than 14 characters lenght.

if you need a vlan tag with long interface name:

you can use "vlan10" interface name with vlan-raw-device

auto vlan10
iface vlan10 inet static
        address ...
        vlan-raw-device enp94s0f0np0

or if you use ifupdown2, you can also use random name

auto mgmt
iface mgmt inet static
        address ..
        vlan-id 10
        vlan-raw-device enp94s0f0np0

(you can manage with gui in last proxmox version)

 

[halley80]

Hello

Thank you for your response.
I understand well for the name of the network legs but I am in a hurry to get back into production.

The network paw names (with Tag Vlan) work on another node.
Do you think I can reintegrate this node without any problem ?

Thank you,
Yours sincerely,

 

[halley80]

Version of PVE => 5.4.13

 

[halley80]

Here is the network interfaces.

What do you think about it?
What are the modifications to be made ?

Thanking you,
Sincerely

 

[spirit]

try

This should fix the management

auto vlan10
iface vlan10 inet manual
        vlan-raw-device enp94s0f0np0

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp94s0f0np0
    bridge-stp off
    bridge-fd 0

auto vmbr0v10
iface vmbr0v10 inet static
    address  192.168.38.108
    netmask  255.255.255.0
    gateway  192.168.38.1
    bridge_ports vlan10
    bridge_stp off
   bridge-fd 0

Also related, you'll have problems, if you use vms defined on vmbr0 with tag=X, as proxmox also try to create vmbr0vX bridges with enp94s0f0np0.X)

A way to get both full working is to use vlan-aware bridge

auto vmbr0
iface vmbr0 inet manual
    bridge_ports enp94s0f0np0
    bridge_stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
    address  192.168.38.108
    netmask  255.255.255.0
    gateway  192.168.38.1

 

[halley80]

Thank you very much.
I will apply this.

However, the cluster communicates on 46.0/24.

Also in the file '/etc/pve/priv/known_hosts' ipmpve6' still appears.
Can this be a problem when adding to the cluster ?

 

[halley80]

Hi,

After a 'fresh_install' of the node.
I applied the network configuration advised/given by 'spirit'.

Many thanks to him.
Best regards,