/etc/pve/nodes/vwsrv3/qemu-server read-only

[hpcraith]

Sorry, I have to post this issue again. This was addressed also by frantek two weeks ago and answered by Udo.
"reach quorum again and your /etc/pve should be writable! .... The answer of frantek was "Thanks. Worked" But he
did not share with us how he managed to reach quorum. I am sure that it is quorum also in my case, but I have no idea
how to reach it. My nodes are more or less up-to-date and I am in the process of updating them one after the other.
What I see:

root@vwsrv2:/boot# pvecm nodes
Node Sts Inc Joined Name
1 M 1756 2015-02-16 11:49:53 vwsrv2
2 M 3608 2015-03-16 13:47:19 vwsrv3
3 M 3580 2015-03-11 11:18:56 vwsrv1
root@vwsrv2:/boot#
root@vwsrv1:/boot# pvecm nodes
Node Sts Inc Joined Name
1 M 3580 2015-03-11 11:18:56 vwsrv2 # pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
2 M 3608 2015-03-16 13:47:19 vwsrv3 # pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
3 M 1876 2015-03-03 18:09:29 vwsrv1 # pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
root@vwsrv1:/boot#
Node Sts Inc Joined Name
1 X 0 vwsrv2
2 M 3600 2015-03-16 13:46:49 vwsrv3
3 X 0 vwsrv1
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server#
vwsrv3 ist the problem node. What does "X" and "M" mean? No description in the manpage.

root@vwsrv2:/boot# pvecm status
Version: 6.2.0
Config Version: 11
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 3
Expected votes: 1
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: vwsrv2
Node ID: 1
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.251
root@vwsrv2:/boot#
root@vwsrv1:/boot# pvecm status
Version: 6.2.0
Config Version: 11
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 3
Expected votes: 1
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 1
Flags:
Ports Bound: 0
Node name: vwsrv1
Node ID: 3
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.248
root@vwsrv1:/boot#
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server# pvecm status
Version: 6.2.0
Config Version: 10
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1 Activity blocked
Active subsystems: 1
Flags: Error
Ports Bound: 0
Node name: vwsrv3
Node ID: 2
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.250
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server#

A very interesting side-effect:
Changing a 101.conf file on vwsrv3 is not possible (read-only)
Changing it on vwsrv1 at root@vwsrv1:/etc/pve/nodes/vwsrv3/qemu-server works and 101.conf file on vwsrv3 is updated.

It would be desirable that proxmox staff collects all these problems and enter it into their wiki as HOWTO.

Thank You

 

[udo]

A very interesting side-effect:
Changing a 101.conf file on vwsrv3 is not possible (read-only)
Changing it on vwsrv1 at root@vwsrv1:/etc/pve/nodes/vwsrv3/qemu-server works and 101.conf file on vwsrv3 is updated.

Hi,
vwsrv3 has problems with the cluster communication, vwsrv1+2 has quorum, this is why you can change there the config (also for node vwsrv3).

Try on vwsrv3 following:

service cman restart
service pvestatd restart
service pve-cluster restart

Udo

 

[hpcraith]

Hi,
vwsrv3 has problems with the cluster communication, vwsrv1+2 has quorum, this is why you can change there the config (also for node vwsrv3).

Try on vwsrv3 following:

service cman restart
service pvestatd restart
service pve-cluster restart

Udo

See output of the commands:

root@vwsrv3:~# service cman restart
Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... Timed-out waiting for cluster
[FAILED]
root@vwsrv3:~#
root@vwsrv3:~# service pvestatd restart
Restarting PVE Status Daemon: pvestatd.
root@vwsrv3:~#

root@vwsrv3:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster[dcdb] crit: local cluster.conf is newer
.
root@vwsrv3:~#

 

[udo]

Restarting pve cluster filesystem: pve-cluster[dcdb] crit: local cluster.conf is newer

Hi,
is your clock (timezone) on all nodes (esp. vwsrv3) correct??

How looks your cluster config (normaly are both the same)

find /etc -name cluster.conf -ls
diff /etc/cluster/cluster.conf /etc/pve/cluster.conf

Udo

 

[hpcraith]

Hi,
is your clock (timezone) on all nodes (esp. vwsrv3) correct??

How looks your cluster config (normaly are both the same)

find /etc -name cluster.conf -ls
diff /etc/cluster/cluster.conf /etc/pve/cluster.conf

Udo

34 1 -r--r----- 1 root www-data 339 Mar 21 2014 /etc/pve/cluster.conf
1794903 4 -rw-r--r-- 1 root root 340 Mar 21 2014 /etc/cluster/cluster.conf
root@vwsrv3:/etc/pve/nodes#
root@vwsrv3:/etc/pve/nodes# diff /etc/cluster/cluster.conf /etc/pve/cluster.conf
2c2
< <cluster name="VW-Cluster" config_version="10">
---
> <cluster name="VW-Cluster" config_version="9">
root@vwsrv3:/etc/pve/nodes#

Dieter

 

[udo]

34 1 -r--r----- 1 root www-data 339 Mar 21 2014 /etc/pve/cluster.conf
1794903 4 -rw-r--r-- 1 root root 340 Mar 21 2014 /etc/cluster/cluster.conf
root@vwsrv3:/etc/pve/nodes#
root@vwsrv3:/etc/pve/nodes# diff /etc/cluster/cluster.conf /etc/pve/cluster.conf
2c2
< <cluster name="VW-Cluster" config_version="10">
---
> <cluster name="VW-Cluster" config_version="9">
root@vwsrv3:/etc/pve/nodes#

Dieter

Hi Dieter,
than copy the file and try again!

cp /etc/pve/cluster.conf /etc/cluster/cluster.conf

Udo

 

[hpcraith]

Hi Dieter,
than copy the file and try again!

cp /etc/pve/cluster.conf /etc/cluster/cluster.conf

Udo

Your suggestion did not work. I edited vwsrv3's /etc/cluster.conf and changed the version number.
Then suddenly I had the quorum needed. But still I could not change the the xxx.conf I wanted to change
on vwsrv3. I repeated the commands
service cman restart
service pvestatd restart
service pve-cluster restart
several times on my 3 nodes.
Suddenly it worked.
Unfortunately I do not understand why.
I would really appreciate a good description of the cluster filesystem.
Dieter

 

[tom]

...
I would really appreciate a good description of the cluster filesystem.
Dieter

start reading here:

http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_(pmxcfs)