Problem with clustering and network on Hetzner hosts during Create/Join cluster procedure

[RubberCluster]

Hello,

I do have 3 identical Hetzner servers that I configured with their Installimage script - Hetzner-Installimage - and configured Network Bridged setup accordingly to their documentation - Hetzner-Proxmox-Network - configuration of network is similar on each host.

Hosts are fresh, no VMs, no containers.
When independent nodes are fully functional, but when I do try to create a cluster I do have the same repetitive problem:
PVE1 - creating cluster do not make any errors, during configuration Link:0 and vmbr0-IPv4 IP were provided, cluster created successfully
Node: PVE1, ID: 1, Votes: 1, Link 0: IPV4
I can pull Join Information but not sure why in the IP Address section IPv6 is listed

PVE2 - joining cluster, while using Assisted Join and providing Join information section Peer Address: is populated with IPv6, providing PVE1 root password, in section Cluster Network: Link 0 is selected as vmbr0 IPV4, Peer's link address : PVE1 IPv4

After accepting configuration and Joining cluster I do have errors:
PVE1 - '/etc/pve/nodes/PVE2/pve-ssl.pem' does not exist! (500)
PVE2 - Taks viewer log:

Establishing API connection with host 'IPv6 of PVE1 here'
Login succeeded.
check cluster join API version
Request addition of this node

permission denied - invalid PVE ticket (401)
PVE1 Datacenter view: PVE1 is GREEN, PVE2 is RED

When changing view on PVE2 (eg accessing Summary) a login window is presented but when providing login (root) and PVE2 password an error is presented:
Login failed, Please try again.

Connection over SSH to PVE2 is still possible with the same credentials without any issues.

PVE2 do not come up with WebUI after restart and PVE1 in Cluster section still show '/etc/pve/nodes/PVE2/pve-ssl.pem' does not exist! (500) even after reboot

I do have the same problem over and over with fresh installations and I just can't handle this anymore...

 

[RubberCluster]

Update #1
Under the WebUI view on PVE1 I tried to access Summary of PVE2 with an error "Connection error 595: No route to host" I had updated /etc/hosts with respective data of each servers

Still no change.

I tried to check mentioned in the error pve-ssl.pem file on PVE2 but is missing (no results from find)
And /etc/pve is empty on PVE2

root@PVE2 /etc/pve # ls -ll
total 1
-r--r----- 1 root www-data 440 Dec 12 11:40 corosync.conf
lr-xr-xr-x 1 root www-data   0 Jan  1  1970 local -> nodes/PVE2
lr-xr-xr-x 1 root www-data   0 Jan  1  1970 lxc -> nodes/PVE2/lxc
lr-xr-xr-x 1 root www-data   0 Jan  1  1970 openvz -> nodes/PVE2/openvz
lr-xr-xr-x 1 root www-data   0 Jan  1  1970 qemu-server -> nodes/PVE2/qemu-server

 

[esi_y]

Since you can SSH in or get pty from both PVE1 and PVE2, can you check (on each separately):

pvecm status
cat /etc/corosync/corosync.conf
cat /etc/network/interfaces
ip -c a
routel

 

[RubberCluster]

Hi @tempacc346235 , thank you for your reply.
Documentation on Hetzner had a problem and it was resolved by their forum member.
Problem was with the wrong netmask for IPv4 and after change from /26 to /32 all of the connection problems were resolved immediately.

 

[pdaniel]

Hello,

I face the same problems, and I cannot find the root cause of this issue. Can you share exactly what you had to change in the network configuration to make it work?

 

[RubberCluster]

Hello,

I face the same problems, and I cannot find the root cause of this issue. Can you share exactly what you had to change in the network configuration to make it work?

Hi, just follow documentation as usual but instead of netmask /26 use /32, it should work out of the box.
Give me a shout if you still face problem, I might reply late but happy to help

https://github.com/hetzneronline/community-content/issues/746

 

[pdaniel]

I have followed the steps from here to install the nodes https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve and setup the interfaces. But when I try to create the cluster I always get the '/etc/pve/nodes/pve2/pve-ssl.pem' does not exist! (500). But the node is connecting to the master because I see it in logs but it gets stucked at waiting for quorum... and after that the error /etc/pve/nodes/pve2/pve-ssl.pem' does not exist! (500) is shown on the master.