Help to fix Proxmox access issues after power cut

[Rollux]

Hi all, long time lurker, first time poster.
Please bear with me a bit, I am seeking assistance to regain access to a proxmox instance - I am a noob as far as Linux and proxmox goes, just playing around with a home lab.

I have 2 old PCs repurposed for home servers (only because I need more RAM do I have 2). I played around with putting them in a cluster, successfully so I thought, and had everything running well for several months.
We had a power cut, and as I have no UPS as yet, the obvious happened. However, something broke in the cluster setup. (FYI - I had rebooted both nodes previously without issue)
What I now can/cannot do:
1. I cannot access Proxmox VE for the "main" server via SSH or web GUI - I get time out errors. However, all VMs and containers are accessible and working as expected.
2. I can access the "secondary" node via its web GUI and SSH - it simply shows the "main" as offline. However, to get the containers and VMs started I need to SSH and advise 'pvecm expected 1' so this node obviously still needs quorum
3. I can ping the "main" server and get a response.

As my main server runs TrueNAS and Immich I don't wish to just wipe it and start again - recovery is preferable but I am at a loss as to how to proceed, despite searching for a few weeks thru various threads on here, reddit and so on.

Any assistance/guidance is much appreciated.

 

[bl1mp]

Hi Rollux,
you could probably try a local login, either via connected screen and keyboard or,
if your server hardware supports it, via out of band management (ILO/IDRAC/BMC ...)

What error message do you get, when you try to connect via ssh -vvv ?

You need to be very careful with changing the expected votes,
this can lead to data loss/split brain for your VM config files and probably also disk data (depending on the setup of the storages),
because they might get overwritten on syncing back to the server, that runs the VMs.
The currently running VM-Processes are not affected directly by that, so do not get confused about that.

A cluster with only two nodes is very fragile, in therms of corosync votes,
because every time one node fails, both nodes will fence themself.
By the reason you also have the Truenas available, I would heavily consider that one
to provide a third vote via qdevice (https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support)

BR, Lucas

 

[proxuser77]

I don't have a “magic bullet” here. But if you can ping it, it seems that at least the network stack is running. To troubleshoot further, I'd say you would need to connect a monitor and keyboard and then see if it gets to the login prompt at startup.

If so, log in as root and check the system logs. If not, see where it stops during the boot process or if you get any errors in the startup logs (the messages that are displayed during the boot process).

EDIT: @bl1mp was faster.

 

[Rollux]

Thanks for the replies.
So I connected monitor and keyboard - can log in ok. But I don't know what to look at or for now. (again, I'm new to this so please be patient)
I checked journalctl -b and only red flags were for a failed CIFS mount (for an LXC) and a Radeon Secure display failure.
I also checked /var/log/pveproxy/ and several of the access logs - can see the various attempts from the IP addresses of the devices on ly network to try and connect

What and where else do I need to look at?