/etc/pve read only / Node cannot be deleted

S

Stephan J.

New Member
Apr 24, 2012
17
1
1
Hello,

We have a 7 node cluster, with 4 servers on Proxmox2 and 3 servers on Proxmox3.3.

We are in the progress of migrating to Promox3.3, thus the mixed environment. So far everything worked fine, but this morning I shut down one of the Proxmox2-nodes and since then /etc/pve is read-only. I cannot remove the node with delnode. I receive an error about I/O error and ressource busy. I checked the forums and quorum does seem to be fine. I tried 'pvecm expected 1', but it did not help.

pvecm delnode s04
Code: 
I/O error : Resource busy
I/O error : Resource busy
ccs_tool: Error writing new config file /etc/pve/cluster.conf

pvecm status:
Code: 
Version: 6.2.0
Config Version: 171
Cluster Name: pve-cluster
Cluster Id: 23476
Cluster Member: Yes
Cluster Generation: 412
Membership state: Cluster-Member
Nodes: 6
Expected votes: 1
Total votes: 6
Node votes: 1
Quorum: 4
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: s01
Node ID: 1
Multicast addresses: 239.192.91.16
Node addresses: 10.10.10.10

clustat
Code: 
Cluster Status for pve-cluster @ Wed Jan 21 14:22:11 2015
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 s01                                                                 1 Online, Local, rgmanager
 s02                                                                 2 Online, rgmanager
 s03                                                                 3 Online, rgmanager
 s04                                                                 4 Offline
 s07                                                                 5 Online, rgmanager
 s08                                                                 6 Online, rgmanager
 s09                                                                 7 Online, rgmanager

 Service Name                                                     Owner (Last)                                                     State
 ------- ----                                                     ----- ------                                                     -----
 service:ha_X1                                                  s03                                                              started
 service:ha_X2                                                  s07                                                              started
 service:ha_X3                                                  s01                                                              started
 service:ha_X4                                                  s07                                                              started
 service:ha_X5                                                  s07                                                              started
 service:ha_X6                                                  s07                                                              started
 service:ha_X7                                                  s02                                                              started
In the Web-GUI, all other nodes are displayed as red.

Can anyone help or give some pointers to a possible cause?

Kind regards -Stephan
 
Thanks for your reply!

It happens on all nodes. /etc/pve is read only on all of them.

Kind regards

-Stephan
 
Starnge. Does it help if you restart pve-cluster:

# service pve-cluster restart

If not, do you get and errors in /var/log/syslog?
 
I got it working.

I executed the command on all nodes and after that 'pvecm status' reported that the cluster was not ready, but clustat reported everything as normal.
After rebooting one of the nodes, it suddenly all worked again and the messages regarding 'rgp_join' in the syslog disappeared.

Thanks!!
 
You must log in or register to reply here.
Top Bottom