fingerprint was mandatory to add a new node to pve 8.1.3 cluster

[akiuni]

Hello

I would like to share an information for other sysadmins who may face the same problem.

I was unable to add a new node (pmox5) to my cluster from CLI, I always get error "500 Can't connect to <cluster_master_ip>:8006 (hostname verification failed)" :

root@pmox5:/var/log# pvecm add pmox1
Please enter superuser (root) password for 'pmox1': ***********
Establishing API connection with host 'pmox1'
The authenticity of host 'pmox1' can't be established.
X509 SHA256 key fingerprint is *****<the fingerprint>***********.
Are you sure you want to continue connecting (yes/no)? yes
500 Can't connect to pmox1:8006 (hostname verification failed)

(Note that all protocols and ports are opened between all the pve nodes and the new one, ssh to pmox1 works fine.)

The solution was to specify the fingerprint, just accepting it was not enought:

root@pmox5:/var/log# pvecm add pmox1 -fingerprint '*****<the fingerprint>***********' 
Please enter superuser (root) password for 'pmox1': ***********
Establishing API connection with host 'pmox1'
Login succeeded.
check cluster join API version
No cluster network links passed explicitly, fallback to local node IP 'pmox5'
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1704714758.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'pmox5' to cluster.

I've never had to do that before and I didn't find this workaround on the other forum posts.

That may be a new feature or maybe bug, I don't know, Here are the differences from past nodes additions (which didn't require the fingerprint):

- old nodes (pmox1-4) have been initially installed on proxmox 7.4.1, the oldest on debian 11 and the latest on debian 12. All have been updated to 8.1.3. Some have been installed from the official iso, and some other from debian packages.
- new node (pmox5) is on a another LAN segment but all traffic is enabled with other nodes.
- new node (pmox5) has been directly installed on proxmox 8.1.3 + debian12 (from the official iso)

 

[tarasp]

I can't thank you enough for pointing out this simple fix. I spent several hours this evening trying to figure this out, but manually specifying -fingerprint did it!

I wonder if the LAN segment thing and possibly something with hostnames is the issue. In my setup, I have a separate VLAN on each machine (in a separate IP address range) that can ping each other no problem on that VLAN, but I ran into this issue when trying to set up a cluster using this VLAN-based IP address range.

 

[esi_y]

Hello

I would like to share an information for other sysadmins who may face the same problem.

I was unable to add a new node (pmox5) to my cluster from CLI, I always get error "500 Can't connect to <cluster_master_ip>:8006 (hostname verification failed)" :

root@pmox5:/var/log# pvecm add pmox1
Please enter superuser (root) password for 'pmox1': ***********
Establishing API connection with host 'pmox1'
The authenticity of host 'pmox1' can't be established.
X509 SHA256 key fingerprint is *****<the fingerprint>***********.
Are you sure you want to continue connecting (yes/no)? yes
500 Can't connect to pmox1:8006 (hostname verification failed)

(Note that all protocols and ports are opened between all the pve nodes and the new one, ssh to pmox1 works fine.)

The solution was to specify the fingerprint, just accepting it was not enought:

root@pmox5:/var/log# pvecm add pmox1 -fingerprint '*****<the fingerprint>***********'
Please enter superuser (root) password for 'pmox1': ***********
Establishing API connection with host 'pmox1'
Login succeeded.
check cluster join API version
No cluster network links passed explicitly, fallback to local node IP 'pmox5'
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1704714758.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'pmox5' to cluster.

I've never had to do that before and I didn't find this workaround on the other forum posts.

That may be a new feature or maybe bug, I don't know, Here are the differences from past nodes additions (which didn't require the fingerprint):

- old nodes (pmox1-4) have been initially installed on proxmox 7.4.1, the oldest on debian 11 and the latest on debian 12. All have been updated to 8.1.3. Some have been installed from the official iso, and some other from debian packages.
- new node (pmox5) is on a another LAN segment but all traffic is enabled with other nodes.
- new node (pmox5) has been directly installed on proxmox 8.1.3 + debian12 (from the official iso)

Did the remote include pmox1 in the SSL certs?

openssl x509 -noout -text -in /etc/pve/local/pve-ssl.pem

Would you (next time maybe?) trypvecm with --use_ssh?

 

[akiuni]

Did the remote include pmox1 in the SSL certs?

openssl x509 -noout -text -in /etc/pve/local/pve-ssl.pem

Would you (next time maybe?) trypvecm with --use_ssh?

I'm sorry but it's too late now because the migration to the target lan is over.

If I make a control now, each local certificate is dedicated to the node it belongs to and doesn't include any information on the other nodes, and they are generated from the same CA. However, I guess that the initial cert you are talking about has been rewritten when I joined the cluster...