[SOLVED] Add new node to cluster

rlljorge

Active Member
Jun 2, 2020
30
4
28
44
Hello,

I'm running a cluster version 6.4 with 7 nodes and I need to add 2 new nodes to the cluster, proxmox08 and proxmox09
I have 2 separate networks for the rings and I added the proxmox08 to the cluster without problem, using the command.

pvecm add proxmox01.sianet.com.br -link0 192.168.30.136 -link1 192.168.30.168

When I tried to add the proxmox09:

pvecm add proxmox01.sianet.com.br -link0 192.168.30.137 -link1 192.168.30.169

The processes stuck in waiting for quorum...
I stopped the process and reinstall the server proxmox09

What I need check for try add node again ?
the proxmox09 can ping all nodes using the rings networks, I have the all servers in /etc/hosts, I have no firewall between the servers, the clock are sync


Code:
root@proxmox08:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-12
pve-kernel-helper: 6.4-12
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1


Code:
root@proxmox08:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 proxmox02
         2          1 proxmox04
         3          1 proxmox03
         4          1 proxmox01
         5          1 proxmox05
         6          1 proxmox06
         7          1 proxmox07
         8          1 proxmox08 (local)

Code:
root@proxmox08:~# pvecm status
Cluster information
-------------------
Name:             sianet
Config Version:   10
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed Feb  9 08:40:58 2022
Quorum provider:  corosync_votequorum
Nodes:            8
Node ID:          0x00000008
Ring ID:          1.6fb
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   8
Highest expected: 8
Total votes:      8
Quorum:           5
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.30.129
0x00000002          1 192.168.30.131
0x00000003          1 192.168.30.130
0x00000004          1 192.168.30.132
0x00000005          1 192.168.30.133
0x00000006          1 192.168.30.134
0x00000007          1 192.168.30.135
0x00000008          1 192.168.30.136 (local)

Any ideas?

Regards,

Rodrigo
 
are there any relevant logs in the syslog/journal from either the cluster or the node to be added?
 
I tried to add the proxmox09 again without success.

pvecm add proxmox01.sianet.com.br -link0 192.168.30.137 -link1 192.168.30.169

A collect the logs bellow.

PROXMOX01 is another member of cluster.

Code:
root@proxmox01:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-02-10 11:38:51 -03; 5 days ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 2186 (corosync)
    Tasks: 9 (limit: 7372)
   Memory: 156.3M
   CGroup: /system.slice/corosync.service
           └─2186 /usr/sbin/corosync -f

Feb 15 16:41:16 proxmox01 corosync[2186]:   [KNET  ] pmtud: PMTUD link change for host: 9 link: 0 from 469 to 1397
Feb 15 16:41:16 proxmox01 corosync[2186]:   [KNET  ] pmtud: PMTUD link change for host: 9 link: 1 from 469 to 1397
Feb 15 16:41:24 proxmox01 corosync[2186]:   [QUORUM] Sync members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:24 proxmox01 corosync[2186]:   [TOTEM ] A new membership (1.7b3) was formed. Members
Feb 15 16:41:24 proxmox01 corosync[2186]:   [QUORUM] Members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:24 proxmox01 corosync[2186]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb 15 16:41:42 proxmox01 corosync[2186]:   [QUORUM] Sync members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:42 proxmox01 corosync[2186]:   [TOTEM ] A new membership (1.7b7) was formed. Members
Feb 15 16:41:42 proxmox01 corosync[2186]:   [QUORUM] Members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:42 proxmox01 corosync[2186]:   [MAIN  ] Completed service synchronization, ready to provide service.

the corosync of proxmox09

Code:
root@proxmox09:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-02-15 16:41:12 -03; 1min 56s ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 41748 (corosync)
    Tasks: 9 (limit: 7372)
   Memory: 148.0M
   CGroup: /system.slice/corosync.service
           └─41748 /usr/sbin/corosync -f

Feb 15 16:41:51 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:41:51 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7b7) was formed. Members
Feb 15 16:42:08 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:08 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7bb) was formed. Members
Feb 15 16:42:25 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:25 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7bf) was formed. Members
Feb 15 16:42:43 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:43 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7c3) was formed. Members
Feb 15 16:43:00 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:43:00 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7c7) was formed. Members

I attached the logs.

Some Idea ? Please Help-me.

Regards,

Rodrigo
 

Attachments

I found.

I didn't have the alias pvelocalhost in /etc/hosts

Before:
Code:
10.1.0.71 proxmox09.sianet.com.br proxmox09

After:
Code:
10.1.0.71 proxmox09.sianet.com.br proxmox09 pvelocalhost

Reference in https://forum.proxmox.com/threads/happen-problem-when-add-a-node-in-cluster-on-pve-7-0.93779/ , Thanks you @hec

Code:
root@proxmox09:~# pvecm nodes


Membership information
----------------------
    Nodeid      Votes Name
         1          1 proxmox02
         2          1 proxmox04
         3          1 proxmox03
         4          1 proxmox01
         5          1 proxmox05
         6          1 proxmox06
         7          1 proxmox07
         8          1 proxmox08
         9          1 proxmox09 (local)

Best Regards,

Rodrigo