[SOLVED] can't convince a part of my nodes to communicate with cluster owner

mocanub · Aug 23, 2023

Hi support / community members,

Since not long ago I've migrated from PVE v7 to v8. Everything seemed fine since a couple of days ago. Now only a part of my 19 total nodes are willing to communicate with the cluster owner and rejoin the cluster.
...

Code:

proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 2.99.0-1
proxmox-backup-file-restore: 2.99.0-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.3
pve-docs: 8.0.3
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.4
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

output for `pvecm status`:

Code:

Cluster information
-------------------
Name:             pve-cluster-is
Config Version:   33
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed Aug 23 15:59:27 2023
Quorum provider:  corosync_votequorum
Nodes:            10
Node ID:          0x00000001
Ring ID:          1.701b
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   19
Highest expected: 19
Total votes:      10
Quorum:           10 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.100.200.23 (local)
0x00000003          1 10.100.50.21
0x00000004          1 10.100.50.24
0x00000005          1 10.100.50.20
0x00000006          1 10.100.50.22
0x00000008          1 10.100.200.11
0x00000009          1 10.100.50.12
0x0000000d          1 10.100.50.17
0x0000000e          1 10.100.50.18
0x00000013          1 10.100.50.29

I know about the different cluster links used by the members. I'm planning to switch them all to 10.100.200.1/24 so that the cluster network traffic is completely separated.

pve-cluster service status on cluster owner:

Code:

root@pve-node-13:/etc/corosync# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-08-23 09:38:21 EEST; 6h ago
    Process: 4786 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 4803 (pmxcfs)
      Tasks: 9 (limit: 154209)
     Memory: 72.6M
        CPU: 23.362s
     CGroup: /system.slice/pve-cluster.service
             └─4803 /usr/bin/pmxcfs

Aug 23 16:32:40 pve-node-13 pmxcfs[4803]: [status] notice: starting data syncronisation
Aug 23 16:32:40 pve-node-13 pmxcfs[4803]: [dcdb] notice: received sync request (epoch 1/4803/00000098)
Aug 23 16:32:40 pve-node-13 pmxcfs[4803]: [status] notice: received sync request (epoch 1/4803/0000006F)
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: received all states
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: leader is 1/4803
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: synced members: 1/4803, 3/2329, 4/2394, 5/2289, 6/2323, 8/29550, 9/145555, 13/3678203, 14/3552, 19/2421
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: start sending inode updates
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: sent all (0) updates
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: all data is up to date
Aug 23 16:33:11 pve-node-13 pmxcfs[4803]: [dcdb] notice: dfsm_deliver_queue: queue length 6

corosync service status on cluster owner:

Code:

root@pve-node-13:/etc/corosync# systemctl status corosync
● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-08-23 09:38:22 EEST; 6h ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 4921 (corosync)
      Tasks: 9 (limit: 154209)
     Memory: 1.6G
        CPU: 2h 25min 31.873s
     CGroup: /system.slice/corosync.service
             └─4921 /usr/sbin/corosync -f

Aug 23 16:34:57 pve-node-13 corosync[4921]:   [TOTEM ] Token has not been received in 17729 ms
Aug 23 16:34:57 pve-node-13 corosync[4921]:   [KNET  ] link: host: 18 link: 0 is down
Aug 23 16:34:57 pve-node-13 corosync[4921]:   [KNET  ] host: host: 18 (passive) best link: 0 (pri: 1)
Aug 23 16:34:57 pve-node-13 corosync[4921]:   [KNET  ] host: host: 18 has no active links
Aug 23 16:35:08 pve-node-13 corosync[4921]:   [KNET  ] rx: host: 18 link: 0 is up
Aug 23 16:35:08 pve-node-13 corosync[4921]:   [KNET  ] link: Resetting MTU for link 0 because host 18 joined
Aug 23 16:35:08 pve-node-13 corosync[4921]:   [KNET  ] host: host: 18 (passive) best link: 0 (pri: 1)
Aug 23 16:35:12 pve-node-13 corosync[4921]:   [KNET  ] rx: host: 17 link: 0 is up
Aug 23 16:35:12 pve-node-13 corosync[4921]:   [KNET  ] link: Resetting MTU for link 0 because host 17 joined
Aug 23 16:35:12 pve-node-13 corosync[4921]:   [KNET  ] host: host: 17 (passive) best link: 0 (pri: 1)

I would really need some guidance on how to troubleshoot and identify my communication issues.

Thanks in advance,
Bogdan

Maximiliano · Aug 24, 2023

Hello could you please share the contents of `/etc/pve/corosync.conf`?

Can the nodes that are in the cluster communicate (ping) with the ones that are not in the cluster?

What about the nodes not in the cluster? Could you share with us the output of `pvecm status` on one of them?

mocanub · Aug 25, 2023

Hi Maximiliano,

Thank you for your reply.

Maximiliano said:
Hello could you please share the contents of `/etc/pve/corosync.conf`?

Sure. This is the corosync.conf file showing on my cluster owner which is pve-node-13:

Code:

root@pve-node-13:/etc/pve/nodes# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-node-01
    nodeid: 8
    quorum_votes: 1
    ring0_addr: 10.100.200.11
  }
  node {
    name: pve-node-02
    nodeid: 9
    quorum_votes: 1
    ring0_addr: 10.100.50.12
  }
  node {
    name: pve-node-03
    nodeid: 10
    quorum_votes: 1
    ring0_addr: 10.100.50.13
  }
  node {
    name: pve-node-05
    nodeid: 12
    quorum_votes: 1
    ring0_addr: 10.100.50.15
  }
  node {
    name: pve-node-06
    nodeid: 20
    quorum_votes: 1
    ring0_addr: 10.100.50.16
  }
  node {
    name: pve-node-07
    nodeid: 13
    quorum_votes: 1
    ring0_addr: 10.100.50.17
  }
  node {
    name: pve-node-08
    nodeid: 14
    quorum_votes: 1
    ring0_addr: 10.100.50.18
  }
  node {
    name: pve-node-09
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.100.50.19
  }
  node {
    name: pve-node-10
    nodeid: 5
    quorum_votes: 1
    ring0_addr: 10.100.50.20
  }
  node {
    name: pve-node-11
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 10.100.50.21
  }
  node {
    name: pve-node-12
    nodeid: 6
    quorum_votes: 1
    ring0_addr: 10.100.50.22
  }
  node {
    name: pve-node-13
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.100.200.23
  }
  node {
    name: pve-node-14
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 10.100.50.24
  }
  node {
    name: pve-node-15
    nodeid: 7
    quorum_votes: 1
    ring0_addr: 10.100.50.25
  }
  node {
    name: pve-node-16
    nodeid: 15
    quorum_votes: 1
    ring0_addr: 10.100.50.26
  }
  node {
    name: pve-node-17
    nodeid: 16
    quorum_votes: 1
    ring0_addr: 10.100.50.27
  }
  node {
    name: pve-node-18
    nodeid: 11
    quorum_votes: 1
    ring0_addr: 10.100.200.28
  }
  node {
    name: pve-node-19
    nodeid: 19
    quorum_votes: 1
    ring0_addr: 10.100.50.29
  }
  node {
    name: pve-node-20
    nodeid: 18
    quorum_votes: 1
    ring0_addr: 10.100.50.30
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: pve-cluster-is
  config_version: 41
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Maximiliano said:
Can the nodes that are in the cluster communicate (ping) with the ones that are not in the cluster?

All the nodes are reachable over the network (they respond to ping).

Maximiliano said:
What about the nodes not in the cluster? Could you share with us the output of `pvecm status` on one of them?

here's the output of pvecm status from one of them (from pve-node-18)

Code:

Cluster information
-------------------
Name:             pve-cluster-is
Config Version:   41
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Aug 25 14:50:33 2023
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x0000000b
Ring ID:          b.83ef
Quorate:          No

Votequorum information
----------------------
Expected votes:   19
Highest expected: 19
Total votes:      1
Quorum:           10 Activity blocked
Flags:            

Membership information
----------------------
    Nodeid      Votes Name
0x0000000b          1 10.100.200.28 (local)

What I've tried last was to remove one of the nodes which was not communicating with the rest of the cluster and tried to forcefully re-add it to the cluster. Was fine shortly after the cluster join but once I restarted that machine it stopped "talking" to the other cluster members.

Again, any advice would be highly appreciated. I've finished my troubleshooting ideas on this one.

Regards,
Bogdan M.

jsterr · Aug 25, 2023

Is this a typo, because the node has 10.100.200.28 instead of 10.100.50.28? Can 10.100.200 communicate withn 10.100.50.X ? Can you show your /etc/hosts of the node that has problems connecting?

mocanub · Aug 25, 2023

jsterr said:
Is this a typo, because the node has 10.100.200.28 instead of 10.100.50.28? Can 10.100.200 communicate withn 10.100.50.X ? Can you show your /etc/hosts of the node that has problems connecting?

Hi jsterr,

I'm trying to separate the management from the cluster traffic. Management traffic for that node is on 10.100.50.28/24 and cluster traffic for that node is being sent via 10.100.200.28/24. Basically, these are 2 Linux VLANs with different tags which rely on the same bridge / NIC (see enp1s0f0 from the attached print screen):

enp1s0f0 and enps0f1 are the ports from a 10Gbit dual port Intel X520-DA2 NIC.

Here's the content of my /etc/hosts files:

Code:

127.0.0.1 localhost.localdomain localhost
10.100.50.28 pve-node-18.symplectic.mgmt pve-node-18

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
# BEGIN ANSIBLE MANAGED: Hostname role
10.100.50.28 pve-node-18.symplectic.mgmt pve-node-18
# END ANSIBLE MANAGED: Hostname role

Regards,
Bogdan M

jsterr · Aug 25, 2023

So does communication between 50.X and 200.X work? Because you use different subnets for corosync as your /etc/pve/corosync.conf states

mocanub · Aug 28, 2023

jsterr said:
So does communication between 50.X and 200.X work? Because you use different subnets for corosync as your /etc/pve/corosync.conf states

Right! ... there wasn't communication between the 2 subnets. Although initially I had a false impression that it did otherwise those nodes should not be able to communicate in first place with the cluster owner and then join the cluster.

Either way, the past weekend I've removed all nodes from the cluster and re-joined them using the 10.100.200.xx links:
pvecm add 10.100.50.18 --link0 10.100.200.xx --force

Thanks @jsterr and @Maximiliano for your help. A lot of good stuff to learn from this forum and from knowledgeable guys like you.

Issue can be marked as solved.

Regards,
Bogdan M.

jsterr · Aug 28, 2023

mocanub said:
View attachment 54649

Right! ... there wasn't communication between the 2 subnets. Although initially I had a false impression that it did otherwise those nodes should not be able to communicate in first place with the cluster owner and then join the cluster.

Either way, the past weekend I've removed all nodes from the cluster and re-joined them using the 10.100.200.xx links:
pvecm add 10.100.50.18 --link0 10.100.200.xx --force

Thanks @jsterr and @Maximiliano for your help. A lot of good stuff to learn from this forum and from knowledgeable guys like you.

Issue can be marked as solved.

Regards,
Bogdan M.

Your welcome! @mocanub you can set the thread to solved by editing it! Please do so

Search

Search

[SOLVED] can't convince a part of my nodes to communicate with cluster owner

mocanub

Active Member

Maximiliano

Proxmox Staff Member

mocanub

Active Member

jsterr

Renowned Member

mocanub

Active Member

jsterr

Renowned Member

mocanub

Active Member

jsterr

Renowned Member