Can't access /etc/pve in failed cluster

[yena]

Hello,
i have a failed 3 node cluster,
two nodes are broken and all vm now are on the node 3.

Now /etc/pve are not mounted and so i hav this error when i try to restart pvestatd:
systemctl restart pvestatd

Oct 04 09:28:37 stgvps pveproxy[1686]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/
Oct 04 09:28:37 stgvps pveproxy[1685]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/
Oct 04 09:28:37 stgvps pveproxy[1687]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/
Oct 04 09:28:37 stgvps pve-ha-lrm[6251]: updating service status from manager failed: Connection refused

If i restart the node, and done: pvecm expect 1
now i can start all VM but afer some minutes web interfaces die and the cluster go offline and again i cant restart pvestatd.

--------------------------------------------------------------------------------------------------------------------------------------------

systemctl restart pve-cluster:

Oct 04 09:35:36 stgvps systemd[1]: pve-cluster.service: Start request repeated too quickly.
Oct 04 09:35:36 stgvps systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'.
Oct 04 09:35:36 stgvps systemd[1]: Failed to start The Proxmox VE cluster filesystem.
-- Subject: A start job for unit pve-cluster.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-cluster.service has finished with a failure.
--
-- The job identifier is 3522 and the job result is failed
-------------------------------------------------------------------------------------------------
systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd
.. No way ..

Any ideas?
Can i remove cluater configuration on the node to use it as a single node ?

Thanks

 

[aaron]

Have a look in our documentation: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_separate_node_without_reinstall

Obviously the steps on the remaining cluster nodes (pvecm delnode, etc.) can be ignored because those nodes are dead anyways.