[SOLVED] Issues with new Node Joins - cluster crashing - fully updated 8.4.1 - no sub

[B-C]

Have a cluster up and running..

over the last year have migrated off vmware as many have.

removed some temporary hosts that were in the cluster with ceph and all going - OK.

Cluster is all by IP vs hostname so cluster chugging along fine - do currently have ceph on the same interface until I can free up some additional ports but all on 10G and doing well.

Issue we've been running into is the cluster when joining a new node freaks out and loses quorum until that new node is unplugged or removed from the network when using the gui to join it.

- First attempt the new node installed ceph but no OSDs configured whoops... figure that might have pissed it off.. drops all storage as the cluster crashes once new node is removed from the network (yank cable) it stabilizes and able to delnode the new node and things are happy healthy again.
/etc/pve/nodes/PVE0<node#>/pve-ssl.pem does not exist! (500)
but does temporarily show up as a node just throws a fit about dns resolution.

- 2nd attempt - reload node same IP but a previous recycled node name - no Ceph installed joining the cluster - same issue (cluster quorum death) Same error
/etc/pve/nodes/PVE0<node#>/pve-ssl.pem does not exist! (500)
but does temporarily show up as a node just throws a fit about dns resolution.
(failed dns lookup or something)

- 3rd attempt - reload node new IP and Nodename - no Ceph installed joining the cluster - same issue (cluster quorum death) Same error
/etc/pve/nodes/PVE0<node#>/pve-ssl.pem does not exist! (500)
same dns oddity

so haven't gone for 4th attempt just yet:

verified a lot of messages about the cluster were pissed in:
journalctl -u pve-cluster -u corosync -b
but not sure what to look for specifically as to why the join is failing.

Added each node name into /etc/hosts for good measure and additionally added into DNS each node
in this case the Node names are capital so all matching /etc/pve/nodes and hostnamectl

cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.99.32 Node02.tld.local Node02 # New Node
192.168.99.33 Node03.tld.local Node03 # Existing
192.168.99.34 Node04.tld.local Node04 # Existing
192.168.99.35 Node05.tld.local Node05 # Existing

Verified from each host they can ssh root@Node0x to each and fingerprint accepted
except Node02 of course but from Node5 I did ssh and verify /etc/hosts all is set to match even though DNS resolution is good - however with DNS servers as VMs the backup hosts file for when things are offline helps for name resolution just incase... unless I setup a dns outside of the cluster.

Any other ideas to cleanup the join process - existing joins were all done via gui... or just join via cli?
currently not confident that the cluster is happy even though currently have green across the board with the 3 node existing cluster.


Something we did notice when joining via the GUI
the new node keeps wanting to use Node05 for example vs the current quorum master listed in the HA.. not sure it matters though...
all appear to have good NTP time and in sync...


Ceph wise has a few customizations:
two different crush rules
one for SSD and another for HDDs to separate the storage

Found initially the pools were blended but separate crush rules seems to have sorted that out... HOPEFULLY that isn't the issue.
---
Really joining node02 should simply join and later start adding OSDs to their respective pools... initially it "should" participate and just use the existing storage.

have other test clusters setup this way in lab enviornment and works fine 4th node has no storage uses NFS / Ceph from the other 3 nodes.

 

[B-C]

issue seems to have been around applied cloudflare cert with the same name to each host...
basically was throwing an odd ssl error on join..

using pvecm add helped to resolve and get through it.
had to use pve.domain.lan to get the join to work correctly ..

issues previously were having HA configured prior to join on a single node whoops... odd part was it should have joined and wiped out any ha settings on the node... why it was blowing up ceph without any ceph config was odd.. but possibly HA wanting to be aware of storage was the issue on that..

either way working as expected now and should have 5th node in by next week.

Currently getting the virtual fortigate moved that uses trunks in vmware > over to Prox bridge a little tricky, and having to configure a dedicated trunk port on hosts so it can just be migrated without all new configuration.