How to manually delete a cluster

[arky_]

Hi,
I had two servers connected into a cluster, but I had to install a PCI Express card in one of them and after starting the server, the servers did not want to connect to one cluster.
When trying to connect to each other, I receive the following error:
detected the following error(s):
* this host already contains virtual guests
TASK ERROR: Check if node may join a cluster failed!
How can I manually remove the cluster information from each server in order to rebuild the cluster?

 

[Chris]

Hi,
I had two servers connected into a cluster, but I had to install a PCI Express card in one of them and after starting the server, the servers did not want to connect to one cluster.
When trying to connect to each other, I receive the following error:
detected the following error(s):
* this host already contains virtual guests
TASK ERROR: Check if node may join a cluster failed!
How can I manually remove the cluster information from each server in order to rebuild the cluster?

Hi,
if you already joined the nodes into a cluster, there is no need to rejoin them again. Just make sure both hosts can reach each other and the pve-cluster service is up and running. Do you see errors in the systemd journal regarding corosync? You can get a paginated view of the journal in reverse chornological order by running journalctl -r -b.

For completeness, the procedure to remove nodes from a cluster is documented here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[arky_]

journalctl -r -b

This command returns me these errors:

-- Journal begins at Wed 2023-06-07 01:47:38 CEST, ends at Fri 2023-11-24 15:05:07 CET. --
Nov 24 15:05:07 proxmox3 sshd[926779]: Connection closed by 10.0.0.121 port 53035
Nov 24 15:05:07 proxmox3 sshd[926779]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:04:52 proxmox3 sshd[926749]: Connection closed by 10.0.0.121 port 53023
Nov 24 15:04:52 proxmox3 sshd[926749]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:04:37 proxmox3 sshd[926698]: Connection closed by 10.0.0.121 port 53010
Nov 24 15:04:37 proxmox3 sshd[926698]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:04:22 proxmox3 sshd[926672]: Connection closed by 10.0.0.121 port 53004
Nov 24 15:04:22 proxmox3 sshd[926672]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:04:07 proxmox3 sshd[926616]: Connection closed by 10.0.0.121 port 52993
Nov 24 15:04:07 proxmox3 sshd[926616]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:03:52 proxmox3 sshd[926588]: Connection closed by 10.0.0.121 port 52983
Nov 24 15:03:52 proxmox3 sshd[926588]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:03:37 proxmox3 sshd[926537]: Connection closed by 10.0.0.121 port 52973
Nov 24 15:03:37 proxmox3 sshd[926537]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:03:22 proxmox3 sshd[926511]: Connection closed by 10.0.0.121 port 52960
Nov 24 15:03:22 proxmox3 sshd[926511]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:03:07 proxmox3 sshd[926455]: Connection closed by 10.0.0.121 port 52948
Nov 24 15:03:07 proxmox3 sshd[926455]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:02:52 proxmox3 sshd[926427]: Connection closed by 10.0.0.121 port 52938
Nov 24 15:02:52 proxmox3 sshd[926427]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:02:37 proxmox3 sshd[926376]: Connection closed by 10.0.0.121 port 52929
Nov 24 15:02:37 proxmox3 sshd[926376]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:02:22 proxmox3 sshd[926350]: Connection closed by 10.0.0.121 port 52917
Nov 24 15:02:22 proxmox3 sshd[926350]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:02:07 proxmox3 sshd[926293]: Connection closed by 10.0.0.121 port 52910
Nov 24 15:02:07 proxmox3 sshd[926293]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:01:52 proxmox3 sshd[926265]: Connection closed by 10.0.0.121 port 52898
Nov 24 15:01:52 proxmox3 sshd[926265]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:01:37 proxmox3 sshd[926214]: Connection closed by 10.0.0.121 port 52885
Nov 24 15:01:37 proxmox3 sshd[926214]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:01:22 proxmox3 sshd[926188]: Connection closed by 10.0.0.121 port 52872
Nov 24 15:01:22 proxmox3 sshd[926188]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:01:07 proxmox3 sshd[926132]: Connection closed by 10.0.0.121 port 52862
Nov 24 15:01:07 proxmox3 sshd[926132]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:00:52 proxmox3 sshd[926103]: Connection closed by 10.0.0.121 port 52855
Nov 24 15:00:52 proxmox3 sshd[926103]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:00:37 proxmox3 sshd[926052]: Connection closed by 10.0.0.121 port 52841
Nov 24 15:00:37 proxmox3 sshd[926052]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:00:22 proxmox3 sshd[926025]: Connection closed by 10.0.0.121 port 52829
Nov 24 15:00:22 proxmox3 sshd[926025]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 15:00:07 proxmox3 sshd[925969]: Connection closed by 10.0.0.121 port 52821
Nov 24 15:00:07 proxmox3 sshd[925969]: error: kex_exchange_identification: Connection closed by remote host
Nov 24 14:59:59 proxmox3 login[925939]: ROOT LOGIN  on '/dev/pts/0'
Nov 24 14:59:59 proxmox3 systemd[1]: Started Session 109 of user root.
Nov 24 14:59:59 proxmox3 systemd[1]: Started User Manager for UID 0.

 

[Chris]

The errors in the logs are only related to sshd, stating that a client with IP 10.0.0.121 closed the connection while trying to connect. Is this the IP of the other node? Please, dump and share the full journal since boot by generating an attachment via journalctl -b > "$(hostname)-journal.txt" for both nodes.

What is the current status of the cluster pvecm status? And what does systemctl status pve-cluster.service tell you on the nodes?

install a PCI Express card in one of them and after starting the server, the servers did not want to connect to one cluster.

The reason this happened is probably that the network interface name for the NIC used as cluster network link changed, therefore a change in /etc/network/interfaces is required. See also https://wiki.debian.org/NetworkInterfaceNames#THE_.22PREDICTABLE_NAMES.22_SCHEME for more information regarding this.

 

[arky_]

1. I ran thejournalctl -b > "$(hostname)-journal.txt" command for both servers and it shows the result -> (attachments).
1.1. IP 10.0.0.121 - This is the address of the virtual machine running "Dude" from Mikrotik, which monitors devices on the network.
1.2. IP 10.0.0.4 - On Proxmox1 This address appears in the logs. This is the address of a Backup server that has long since been disconnected.
2. pvecm status:

Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?

3. systemctl status pve-cluster.service:

Error: Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
root@proxmox3:~# systemctl status pve-cluster.service
 pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-11-20 16:26:15 CET; 4 days ago
    Process: 2333 ExecStart=/usr/bin/pmxcfs (code=exited, status=0/SUCCESS)
   Main PID: 2352 (pmxcfs)
      Tasks: 10 (limit: 38441)
     Memory: 71.5M
        CPU: 4min 36.835s
     CGroup: /system.slice/pve-cluster.service
             └─2352 /usr/bin/pmxcfs

Nov 20 16:26:14 proxmox3 systemd[1]: Starting The Proxmox VE cluster filesystem...
Nov 20 16:26:15 proxmox3 systemd[1]: Started The Proxmox VE cluster filesystem.

I don't want to find out why they disconnected and how to connect the same cluster, I just want to manually remove the cluster on both sides and only then reconnect these two servers into one cluster.

 

[Chris]

I had two servers connected into a cluster

According to the output of pvecm status of node proxmox3 you do not have a cluster, so your comments do not reflect the state as is. What does the same command show on node proxmox1? Also, note that you cannot add a host to a cluster if it already contains VMs/CTs, you will have to have an empty node, please follow the docs carefully https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_join_node_to_cluster

P 10.0.0.4 - On Proxmox1 This address appears in the logs. This is the address of a Backup server that has long since been disconnected.

Please remove the storage related to the backup server if it is no longer available. Optionally, if it will become available again in the future, you can also set it to disabled by unchecking the enabled checkbox in Datacenter > Storage > <PBS Storage> > Edit. This should get rid of the errors in the logs.

1. IP 10.0.0.121 - This is the address of the virtual machine running "Dude" from Mikrotik, which monitors devices on the network.

Is it intended for this service to try to connect to the host via ssh and then drop the connection?