Can't remove node from cluster. Whole series of errors

[Gorf]

I have a faulty node that I am going to remove entirely to decommission. But I'm totally stuck doing so. I started pretty simply...

root@pve:~# pvecm delnode utility1
trying to acquire cfs lock 'file-corosync_conf' ...
Killing node 2
unable to open file '/etc/pve/corosync.conf.new.tmp.2910428' - Permission denied

That tmp file doesn't exist and doesn't appear to ever exist as far as I can tell watching the directory during the process. I tried to stop the corosynce process and edit the file directly but even with root I'm getting permission denied and so I assume there are attribs or something. pvecm status only shows one node, BUT the GUI still shows the ejected node but greyed out and with a red X on it. The documentation says the last step is to remove the nodes folder but that doesn't work:

root@pve:/etc/pve# rm -rf nodes/utility1
rm: cannot remove 'nodes/utility1/lxc': Permission denied
rm: cannot remove 'nodes/utility1/pve-ssl.key': Permission denied
rm: cannot remove 'nodes/utility1/lrm_status': Permission denied
rm: cannot remove 'nodes/utility1/pve-ssl.pem': Permission denied
rm: cannot remove 'nodes/utility1/priv': Permission denied
rm: cannot remove 'nodes/utility1/ssh_known_hosts': Permission denied
rm: cannot remove 'nodes/utility1/openvz': Permission denied
rm: cannot remove 'nodes/utility1/qemu-server': Permission denied


So what gives here? How do you remove a node? Because the documentation just simply doesn't work.

 

[UdoB]

Permission denied

You can only administrate a cluster (by writing to /etc/pve implicitly) while Quorum is given.

What is your situation? Examine pvecm status.

If you have only one node left and want to destroy an "old" second one: you may force the remaining node to lie and set pvecm expected 123 - please read man pvecm first.

Then you can "rm" all residuals of the old "utility1" from /etc/pve and below.

Disclaimer: be careful and have a backup - here be dragons!