Inexperienced Admins broke our Proxmox node, help!

[lightweb]

So, some other, less experienced admins decided it was a good idea to try and cluster our proxmox nodes without actually understanding how to do it. We do not have shared storage and of course it failed but they left it in a half configured state.

Someone just rebooted one of the nodes and now all of the configuration files are gone from /etc/qemu-server and /etc/openvz. The fuse directory is mounted under /etc/pve but the /etc/pve/nodes/[hostname]/... is empty.

----
/etc/pve/cluster.conf:

<?xml version="1.0"?>
<cluster name="bitvps" config_version="10">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey">
</cman>
<clusternodes>
</clusternodes>
</cluster>

----

service cman start fails with:
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... Cannot find node name in cluster.conf
Unable to get the configuration
Cannot find node name in cluster.conf
cman_tool: corosync daemon didn't start Check cluster logs for details
[FAILED]

----

service pve-cluster start works and like I said, /etc/pve seems mounted.


I've backed up /etc/pve/*


What can I do to remove the cluster configuration and get us back on track? Anything?

We have both openvz and KVMs on this host and I guess they appear to be running ok. I can not execute qm list or vzlist.


Help!

Thanks,
lw

 

[udo]

So, some other, less experienced admins decided it was a good idea to try and cluster our proxmox nodes without actually understanding how to do it. We do not have shared storage and of course it failed but they left it in a half configured state.

Someone just rebooted one of the nodes and now all of the configuration files are gone from /etc/qemu-server and /etc/openvz. The fuse directory is mounted under /etc/pve but the /etc/pve/nodes/[hostname]/... is empty.

----
/etc/pve/cluster.conf:

<?xml version="1.0"?>
<cluster name="bitvps" config_version="10">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey">
</cman>
<clusternodes>
</clusternodes>
</cluster>

----

service cman start fails with:
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... Cannot find node name in cluster.conf
Unable to get the configuration
Cannot find node name in cluster.conf
cman_tool: corosync daemon didn't start Check cluster logs for details
[FAILED]

----

service pve-cluster start works and like I said, /etc/pve seems mounted.


I've backed up /etc/pve/*


What can I do to remove the cluster configuration and get us back on track? Anything?

We have both openvz and KVMs on this host and I guess they appear to be running ok. I can not execute qm list or vzlist.


Help!

Thanks,
lw

Hi,
your content of /etc/pve can't be the right if the nodes was before in the cluster.
Look in /var/lib/pve-cluster and compare the content with an running cluster-node.

Has the node the right nodename? And the right entrys in /etc/hosts? IP-Adreses?

If you have the right content in /var/lib/pve-cluster you can use "pmxcfs --local" to mount the /etc/pve-content without cluster.

Udo

 

[lightweb]

Hi,
your content of /etc/pve can't be the right if the nodes was before in the cluster.
Look in /var/lib/pve-cluster and compare the content with an running cluster-node.

Has the node the right nodename? And the right entrys in /etc/hosts? IP-Adreses?

If you have the right content in /var/lib/pve-cluster you can use "pmxcfs --local" to mount the /etc/pve-content without cluster.

Udo

Thank you for the help. Someone else came in and changed something and now it's fixed. I'm not sure if they used your suggestion or not.

No change control -- What a mess. I think I'm removing myself from the project.

Thanks again.