[SOLVED] Node not in cluster after debian update

[anone]

Hello everyone,

first time post, sorry if the formatting is wrong.

So I started in a company and took over a proxmox cluster on Debian 11. After toying with proxmox for a few months, I decided to check the last thing on my predecessor's to-do list "keep the system up to date".

Simple enough task right ? I checked a few guides and ended up doing an apt-get update && apt-get upgrade ( I, of course turned off all the VM's and LXC's before hand).
Upgrade goes through, I still have some "pve-kernel" updates to do but I want to check things first. I reload my WEB UI, and there it was, my node1 could only see node2 (I have 6 in total) and he saw it offline. I check the logs and it says something along the lines of can't find hostname. I edit my /etc/hosts and it's working fine again (no green icon on the node tho ?). I repeat the process for the nodes left, but nothing.

When I log on my second node's web UI, all the nodes are there, but the first node has a red icon saying it's unavailable.

If i try to start a VM or LXC on node1 i get "cluster not ready - no quorum? (500)".

The .members files on node1 is basically empty, while good on node2. The corosyncs files are exactly the same.

Pvecm status returns "Cannot initialize CMAP service".

On node2, corosync status says that host 4 (node1) has no active links (all my nodes can ping one another).

I am on proxmox 7.2-7 and havent upgraded to 7.3 yet (was planning on it ... ).

You will find screenshots of everything with my post.

Thank you for your time and effort.

Friendly yours,

Anone

 

[Moayad]

Simple enough task right ? I checked a few guides and ended up doing an apt-get update && apt-get upgrade ( I, of course turned off all the VM's and LXC's before hand).

We highly recommend to not use the (apt update && apt upgrade) instead you have to do (apt update && apt dist-upgrade).

Can you please provide us with the Corosync config as well as the output of pveversion -v

 

[anone]

Thanks for the quick response, I didn't know how you wanted it so I put the results in txt files.

Friendly yours,

Anone

 

[Moayad]

Hello,

Thank you for the files!

Can you please do (apt update && apt dist-upgrade) if this does not help please provide us with the Syslog (/var/log/syslog) as attach.

 

[anone]

well I clearly forgot to mention, I dont have access to internet anymore. The place I work at doesn't allow internet access on the server and I only had internet via a VM who's MAC was allowed to go on the web (don't ask me why, it's managed by another team). I tried the apt dist-upgrade but It can't reach internet. I can try via the web ui tho as I have these to updates :



Sorry I didn't mention this sooner.

Friendly yours,

Anone

 

[anone]

Otherwise been thinking of doing :

pvecm expected 1;
pvecm add IP-OF-ANY-CLUSTER-NODE;

Would that work out ? What about the pending updates tho ?

Friendly yours,

Anone

 

[Moayad]

Hello,

Can you please provide us with the Syslog to investigate the issue?

What about the pending updates tho ?

Since the nodes don't have internet, you can use Proxmox Offline Mirror to do the right upgrade.

pvecm expected 1;
pvecm add IP-OF-ANY-CLUSTER-NODE;

Would that work out ?

No.

 

[anone]

Hello, (thank you again btw)

I dont think I was clear enough, I don't have an internet access at all. Only the MAC of a VM's NIC was allowed internet access but I can't start it anymore so I cant do "apt install proxmox-offline-mirror".

You will find the syslog file attached. (I had to delete most of if because it was too big to upload, either way it's the same error messages in a loop).

When running status corosyn on node2 for example, I get :

I don't understand why I get host 4 has no active links since I just updated&upgraded (did not touch the network settings) and they can still ping each other.


Friendly Yours,


Anone

 

[anone]

edit : I just restarted coroync and pve-cluster on node2 and now it has the same error message as my first post, meaning node2 only sees node1 offline and doesn't see the other nodes anymore.

 

[anone]

second edit : I managed to finish the dist upgrade, so I now run proxmox 7.3-3 ( I had a hard time with pve-manager ) but it still doesn't work.

Syslog stil says Cluster not quorate -- extending auth key lifetime!

 

[Moayad]

Hello,

Can you please provide us with full syslog since the upgrade?

After you did the upgrade to 7.3 did you try to restart the pve-cluster and corosync services?

 

[anone]

Hello,

yes, I restarted both services which gave me the same results as before.

what I don't understand is when I look at the corosync status on node3 it says that host link is down :



I'll attach the syslogs of today.

Friendly yours,

Anone

 

[Moayad]

Hello,

Thank you for the syslog!

Can the nodes ping each other on the subnet `192.168.100.x`? can you please also provide us with the command corosync-cfgtool -s output

Do the nodes uses the PVE repository [0]?

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories

 

[anone]

Yes, all the nodes can ping each other.

The command :



Repo :



Friendly yours,

Anone

 

[Moayad]

Hello,

Is the output of the `corosync-cfgtool -s` command the same on all nodes?

 

[anone]

oops sorry, here goes :



I get this on all other nodes.

Friendly yours,

Anone

 

[Moayad]

All the nodes are running with the same version of corosync: 3.1.7-pve1 right?


Note: we recommend having separate NIC only for the Corosync or/and adding a redundant ring as described in our Docs guide [0]


[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[anone]

I just checked the version and no, node1 has 3.1.7 (which makes sense since it's the one i updated) but the rest has 3.1.5 (which doesnt explain why node2 left the cluster too).

Concerning the NIC's, when I joined the company everything was already set up, VM's and LXC's were running and I've been working on them for the past 3 months, and still learning/discovering about proxmox.

I am willing to try and set up a redundant ring once I get this back up !

Friendly yours,

Anone

 

[Moayad]

Hi,

Consider upgrading the nodes to be in the same version, especially the Corosync in order to narrow down the issue.

 

[anone]

Hi,

Consider upgrading the nodes to be in the same version, especially the Corosync in order to narrow down the issue.

ok so just to make sure you want me to do apt update && apt dist-upgrade on all my nodes ?

Friendly yours,

Anone