[SOLVED] Add new node to cluster

rlljorge

Member
Jun 2, 2020
30
4
13
43
Hello,

I'm running a cluster version 6.4 with 7 nodes and I need to add 2 new nodes to the cluster, proxmox08 and proxmox09
I have 2 separate networks for the rings and I added the proxmox08 to the cluster without problem, using the command.

pvecm add proxmox01.sianet.com.br -link0 192.168.30.136 -link1 192.168.30.168

When I tried to add the proxmox09:

pvecm add proxmox01.sianet.com.br -link0 192.168.30.137 -link1 192.168.30.169

The processes stuck in waiting for quorum...
I stopped the process and reinstall the server proxmox09

What I need check for try add node again ?
the proxmox09 can ping all nodes using the rings networks, I have the all servers in /etc/hosts, I have no firewall between the servers, the clock are sync


Code:
root@proxmox08:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.162-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-12
pve-kernel-helper: 6.4-12
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1


Code:
root@proxmox08:~# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 proxmox02
         2          1 proxmox04
         3          1 proxmox03
         4          1 proxmox01
         5          1 proxmox05
         6          1 proxmox06
         7          1 proxmox07
         8          1 proxmox08 (local)

Code:
root@proxmox08:~# pvecm status
Cluster information
-------------------
Name:             sianet
Config Version:   10
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed Feb  9 08:40:58 2022
Quorum provider:  corosync_votequorum
Nodes:            8
Node ID:          0x00000008
Ring ID:          1.6fb
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   8
Highest expected: 8
Total votes:      8
Quorum:           5
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.30.129
0x00000002          1 192.168.30.131
0x00000003          1 192.168.30.130
0x00000004          1 192.168.30.132
0x00000005          1 192.168.30.133
0x00000006          1 192.168.30.134
0x00000007          1 192.168.30.135
0x00000008          1 192.168.30.136 (local)

Any ideas?

Regards,

Rodrigo
 
are there any relevant logs in the syslog/journal from either the cluster or the node to be added?
 
I tried to add the proxmox09 again without success.

pvecm add proxmox01.sianet.com.br -link0 192.168.30.137 -link1 192.168.30.169

A collect the logs bellow.

PROXMOX01 is another member of cluster.

Code:
root@proxmox01:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-02-10 11:38:51 -03; 5 days ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 2186 (corosync)
    Tasks: 9 (limit: 7372)
   Memory: 156.3M
   CGroup: /system.slice/corosync.service
           └─2186 /usr/sbin/corosync -f

Feb 15 16:41:16 proxmox01 corosync[2186]:   [KNET  ] pmtud: PMTUD link change for host: 9 link: 0 from 469 to 1397
Feb 15 16:41:16 proxmox01 corosync[2186]:   [KNET  ] pmtud: PMTUD link change for host: 9 link: 1 from 469 to 1397
Feb 15 16:41:24 proxmox01 corosync[2186]:   [QUORUM] Sync members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:24 proxmox01 corosync[2186]:   [TOTEM ] A new membership (1.7b3) was formed. Members
Feb 15 16:41:24 proxmox01 corosync[2186]:   [QUORUM] Members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:24 proxmox01 corosync[2186]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb 15 16:41:42 proxmox01 corosync[2186]:   [QUORUM] Sync members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:42 proxmox01 corosync[2186]:   [TOTEM ] A new membership (1.7b7) was formed. Members
Feb 15 16:41:42 proxmox01 corosync[2186]:   [QUORUM] Members[8]: 1 2 3 4 5 6 7 8
Feb 15 16:41:42 proxmox01 corosync[2186]:   [MAIN  ] Completed service synchronization, ready to provide service.

the corosync of proxmox09

Code:
root@proxmox09:~# systemctl status corosync
● corosync.service - Corosync Cluster Engine
   Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-02-15 16:41:12 -03; 1min 56s ago
     Docs: man:corosync
           man:corosync.conf
           man:corosync_overview
 Main PID: 41748 (corosync)
    Tasks: 9 (limit: 7372)
   Memory: 148.0M
   CGroup: /system.slice/corosync.service
           └─41748 /usr/sbin/corosync -f

Feb 15 16:41:51 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:41:51 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7b7) was formed. Members
Feb 15 16:42:08 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:08 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7bb) was formed. Members
Feb 15 16:42:25 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:25 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7bf) was formed. Members
Feb 15 16:42:43 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:42:43 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7c3) was formed. Members
Feb 15 16:43:00 proxmox09 corosync[41748]:   [QUORUM] Sync members[1]: 9
Feb 15 16:43:00 proxmox09 corosync[41748]:   [TOTEM ] A new membership (9.7c7) was formed. Members

I attached the logs.

Some Idea ? Please Help-me.

Regards,

Rodrigo
 

Attachments

  • proxmox01.txt
    12.4 KB · Views: 1
  • proxmox09.txt
    36.3 KB · Views: 0
I found.

I didn't have the alias pvelocalhost in /etc/hosts

Before:
Code:
10.1.0.71 proxmox09.sianet.com.br proxmox09

After:
Code:
10.1.0.71 proxmox09.sianet.com.br proxmox09 pvelocalhost

Reference in https://forum.proxmox.com/threads/happen-problem-when-add-a-node-in-cluster-on-pve-7-0.93779/ , Thanks you @hec

Code:
root@proxmox09:~# pvecm nodes


Membership information
----------------------
    Nodeid      Votes Name
         1          1 proxmox02
         2          1 proxmox04
         3          1 proxmox03
         4          1 proxmox01
         5          1 proxmox05
         6          1 proxmox06
         7          1 proxmox07
         8          1 proxmox08
         9          1 proxmox09 (local)

Best Regards,

Rodrigo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!