An error occured on the cluster node: corosync: link 1 exists in cluster config but wasn't specified for new node

[Matthias Looss]

HI,

Looking for some help with adding 4th node to cluster.

I have a 3 node cluster and I am trying to add another node. My 3 node cluster was running PVE6 and I successfully upgrade each node to PVE7 with no issues. The cluster remained intact and has full quorum. I prepared a new node with a fresh install of PVE7 and I am trying now to join this node to the existing cluster. All nodes run the same and the latest versions 7.0-11.

When I copy and paste the cluster info into the new node using Assisted join, it populates all fields except the password and two link fields. The link fields each show the same IP address for the peers link address and the Join PVE-Cluster button is not active.

When I uncheck assisted join, the add button becomes active but throws the following error.

This is my own personal home-lab for running some basic VMs and Containers, and learning all I can about PROXMOX.

Thanks in advance!

Establishing API connection with host '10.0.1.244'
Login succeeded.
check cluster join API version
Request addition of this node
An error occured on the cluster node: corosync: link 1 exists in cluster config but wasn't specified for new node
TASK ERROR: Cluster join aborted!

 

[fiona]

Hi,
please share the output of /etc/pve/corosync.conf for a node in the cluster and pveversion -v for both, the cluster node and the new node.

 

[Matthias Looss]

logging {
  debug: off
  to_syslog: yes
}


nodelist {
  node {
    name: pve-hn1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.1.243
    ring1_addr: 10.0.1.243
  }
  node {
    name: pve-hn2
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.0.1.244
    ring1_addr: 10.0.1.244
  }
  node {
    name: pve-hn3
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.0.1.245
    ring1_addr: 10.0.1.245
  }
}


quorum {
  provider: corosync_votequorum
}


totem {
  cluster_name: PVE-Cluster
  config_version: 3
  interface {
    linknumber: 0
  }
  interface {
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

 

[Matthias Looss]

And here is the version info:

root@pve-hn2:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-4-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-7
pve-kernel-helper: 7.0-7
pve-kernel-5.4: 6.4-5
pve-kernel-5.3: 6.1-6
pve-kernel-5.11.22-4-pve: 5.11.22-8
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.3-1
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

 

[fiona]

nodelist {
  node {
    name: pve-hn1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.0.1.243
    ring1_addr: 10.0.1.243

Different rings are used as fallbacks for redundancy, so using the same IP for both won't give you anything. Except maybe problems when joining a new node

You could try joining the node via CLI, using --link0 <IP of new node> --link1 <IP of new node> options for the pvecm command, but I'd say fixing your configuration is still a good idea. See here for how to edit the corosync configuration and best to follow each step closely.

 

[Matthias Looss]

Ok, not sure how this second link got in there, my cluster has been working fine for a long time. I manually edit the corosync.conf by removing the references to it on all nodes. I had to manually restart the corosync but after that, I was able to join the cluster. Except that the new node shows offline in the cluster web GUI now. While troubleshooting this issue, I now noticed that all my nodes are oflfine in the web GUI, but I can ssh into all of them?

The quorum now shows 3 activity blocked?

root@pve-hn4:~# pvecm status
Cluster information
-------------------
Name:             PVE-Cluster
Config Version:   4
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Sep 14 09:43:22 2021
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000004
Ring ID:          4.c3d7
Quorate:          No

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      1
Quorum:           3 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000004          1 10.0.1.246 (local)
root@pve-hn4:~# pvecm status
Cluster information
-------------------
Name:             PVE-Cluster
Config Version:   4
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Sep 14 10:19:56 2021
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000004
Ring ID:          4.d457
Quorate:          No

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      1
Quorum:           3 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000004          1 10.0.1.246 (local)
root@pve-hn4:~# ls -l
total 0
root@pve-hn4:~#

 

[Matthias Looss]

I rebooted each node and now they are showing up in the web GUI, and all looks to be working.