Unable to join cluster 7.4-3

[mbabin]

Hello I am new to Proxmox so let me know if I miss something obvious.
I have 4 nodes freshly installed with 7.4-3. I did the community updates. They are in a Dell VRTX but that shouldn't matter.
The NICs are all bonded and communicating. I can ping and ssh into all of them with name and IP.
pve 10.0.200.60
pve1 10.0.200.61
pve2 10.0.200.62
pve3 10.0.200.63

I created a cluster, VRTX, on pve. I cannot join the cluster from any other node. I get:

Establishing API connection with host '10.0.200.60'
TASK ERROR: 500 Can't connect to 10.0.200.60:8006

I tried to remove the cluster and add it again but I get the same results.
What should I try?

 

[bbgeek17]

From the node where you are trying to join, what happens when you :
a) ping 10.0.200.60
b) curl -sk https://10.0.200.60:8006

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[mbabin]

The ping works and the curl errors out.

 

[bbgeek17]

initiate "journalctl -f" on the primary and secondary node, then examine its context after you attempt to join.
Try to use CLI, instead of GUI: https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_join_node_to_cluster
Provide output of each command listed there in CODE tags.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Saahib]

Also, disable firewall if there is any while you are trying to connect to cluster, you may later enable it.

 

[mbabin]

on PVE

root@pve:~# journalctl -f
-- Journal begins at Wed 2023-05-03 15:38:42 EDT. --
May 05 15:33:23 pve systemd[47757]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
May 05 15:33:23 pve systemd[47757]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
May 05 15:33:23 pve systemd[47757]: Listening on GnuPG cryptographic agent and passphrase cache.
May 05 15:33:23 pve systemd[47757]: Reached target Sockets.
May 05 15:33:23 pve systemd[47757]: Reached target Basic System.
May 05 15:33:23 pve systemd[47757]: Reached target Main User Target.
May 05 15:33:23 pve systemd[47757]: Startup finished in 94ms.
May 05 15:33:23 pve systemd[1]: Started User Manager for UID 0.
May 05 15:33:23 pve systemd[1]: Started Session 16 of user root.
May 05 15:33:23 pve login[47772]: ROOT LOGIN  on '/dev/pts/0'

on PVE1

root@pve1:~# journalctl -f
-- Journal begins at Thu 2023-05-04 10:40:12 EDT. --
May 05 15:33:54 pve1 systemd[47827]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
May 05 15:33:54 pve1 systemd[47827]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
May 05 15:33:54 pve1 systemd[47827]: Listening on GnuPG cryptographic agent and passphrase cache.
May 05 15:33:54 pve1 systemd[47827]: Reached target Sockets.
May 05 15:33:54 pve1 systemd[47827]: Reached target Basic System.
May 05 15:33:54 pve1 systemd[47827]: Reached target Main User Target.
May 05 15:33:54 pve1 systemd[47827]: Startup finished in 108ms.
May 05 15:33:54 pve1 systemd[1]: Started User Manager for UID 0.
May 05 15:33:54 pve1 systemd[1]: Started Session 13 of user root.
May 05 15:33:54 pve1 login[47841]: ROOT LOGIN  on '/dev/pts/0'
May 05 15:34:32 pve1 sshd[47910]: Accepted password for root from 10.0.100.114 port 53071 ssh2
May 05 15:34:32 pve1 sshd[47910]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 05 15:34:32 pve1 systemd-logind[1270]: New session 15 of user root.
May 05 15:34:32 pve1 systemd[1]: Started Session 15 of user root.
May 05 15:34:55 pve1 pvecm[47987]: <root@pam> starting task UPID:pve1:0000BB74:001F7754:64555A5F:clusterjoin::root@pam:
May 05 15:36:00 pve1 pvecm[47988]: 500 Can't connect to 10.0.200.60:8006
May 05 15:36:00 pve1 pvecm[47987]: <root@pam> end task UPID:pve1:0000BB74:001F7754:64555A5F:clusterjoin::root@pam: 500 Can't connect to 10.0.200.60:8006

 

[bbgeek17]

As @Saahib mentioned, ensure there is no erroneous firewall rule. Check MTU sizes, are you using Jumbo by chance? Double check for duplicate IPs.
Beyond that - get tcpdump running on both sides capturing traffic to/from port 8006 and see if there is anything interesting.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[mbabin]

It was the Jumbo MTU. It set it all back to 1500 and they joined!!

Thanks!!!

 

[agniroot]

has anyone managed to attach a node to a cluster on mtu 9000? It's just a lot of notes and it's unrealistic to switch everyone to 1500. at the same time, packets from mtu 9000 pass successfully ping -s 8972 -M do <ip> But is it half-connected to the cluster? Can't get the keys /etc/pve/nodes/ds1111111/pve-ssl.pem' does not exist! (500)

Establishing API connection with host '11.1.1.1'
Login succeeded.
check cluster join API version
No cluster network links passed explicitly, fallback to local node IP '11.1.1.2'
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service