/etc/pve/nodes/vwsrv3/qemu-server read-only

H

hpcraith

Renowned Member
Proxmox Subscriber
Mar 8, 2013
84
0
71
Stuttgart Germany
www.hlrs.de
Sorry, I have to post this issue again. This was addressed also by frantek two weeks ago and answered by Udo.
"reach quorum again and your /etc/pve should be writable! .... The answer of frantek was "Thanks. Worked" But he
did not share with us how he managed to reach quorum. I am sure that it is quorum also in my case, but I have no idea
how to reach it. My nodes are more or less up-to-date and I am in the process of updating them one after the other.
What I see:

root@vwsrv2:/boot# pvecm nodes
Node Sts Inc Joined Name
1 M 1756 2015-02-16 11:49:53 vwsrv2
2 M 3608 2015-03-16 13:47:19 vwsrv3
3 M 3580 2015-03-11 11:18:56 vwsrv1
root@vwsrv2:/boot#
root@vwsrv1:/boot# pvecm nodes
Node Sts Inc Joined Name
1 M 3580 2015-03-11 11:18:56 vwsrv2 # pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
2 M 3608 2015-03-16 13:47:19 vwsrv3 # pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
3 M 1876 2015-03-03 18:09:29 vwsrv1 # pve-manager: 3.4-1 (running version: 3.4-1/3f2d890e)
root@vwsrv1:/boot#
Node Sts Inc Joined Name
1 X 0 vwsrv2
2 M 3600 2015-03-16 13:46:49 vwsrv3
3 X 0 vwsrv1
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server#
vwsrv3 ist the problem node. What does "X" and "M" mean? No description in the manpage.

root@vwsrv2:/boot# pvecm status
Version: 6.2.0
Config Version: 11
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 3
Expected votes: 1
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: vwsrv2
Node ID: 1
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.251
root@vwsrv2:/boot#
root@vwsrv1:/boot# pvecm status
Version: 6.2.0
Config Version: 11
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 3
Expected votes: 1
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 1
Flags:
Ports Bound: 0
Node name: vwsrv1
Node ID: 3
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.248
root@vwsrv1:/boot#
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server# pvecm status
Version: 6.2.0
Config Version: 10
Cluster Name: VW-Cluster
Cluster Id: 17844
Cluster Member: Yes
Cluster Generation: 3608
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1 Activity blocked
Active subsystems: 1
Flags: Error
Ports Bound: 0
Node name: vwsrv3
Node ID: 2
Multicast addresses: 239.192.69.249
Node addresses: xxx.xx.x.250
root@vwsrv3:/etc/pve/nodes/vwsrv3/qemu-server#

A very interesting side-effect:
Changing a 101.conf file on vwsrv3 is not possible (read-only)
Changing it on vwsrv1 at root@vwsrv1:/etc/pve/nodes/vwsrv3/qemu-server works and 101.conf file on vwsrv3 is updated.

It would be desirable that proxmox staff collects all these problems and enter it into their wiki as HOWTO.

Thank You
 
hpcraith said:
A very interesting side-effect:
Changing a 101.conf file on vwsrv3 is not possible (read-only)
Changing it on vwsrv1 at root@vwsrv1:/etc/pve/nodes/vwsrv3/qemu-server works and 101.conf file on vwsrv3 is updated.
Click to expand...
Hi,
vwsrv3 has problems with the cluster communication, vwsrv1+2 has quorum, this is why you can change there the config (also for node vwsrv3).

Try on vwsrv3 following:
Code: 
service cman restart
service pvestatd restart
service pve-cluster restart
Udo
 
udo said:
Hi,
vwsrv3 has problems with the cluster communication, vwsrv1+2 has quorum, this is why you can change there the config (also for node vwsrv3).

Try on vwsrv3 following:
Code: 
service cman restart
service pvestatd restart
service pve-cluster restart
Udo
Click to expand...

See output of the commands:

root@vwsrv3:~# service cman restart
Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... Timed-out waiting for cluster
[FAILED]
root@vwsrv3:~#
root@vwsrv3:~# service pvestatd restart
Restarting PVE Status Daemon: pvestatd.
root@vwsrv3:~#

root@vwsrv3:~# service pve-cluster restart
Restarting pve cluster filesystem: pve-cluster[dcdb] crit: local cluster.conf is newer
.
root@vwsrv3:~#
 
udo said:
Hi,
is your clock (timezone) on all nodes (esp. vwsrv3) correct??

How looks your cluster config (normaly are both the same)
Code: 
find /etc -name cluster.conf -ls
diff /etc/cluster/cluster.conf /etc/pve/cluster.conf
Udo
Click to expand...

34 1 -r--r----- 1 root www-data 339 Mar 21 2014 /etc/pve/cluster.conf
1794903 4 -rw-r--r-- 1 root root 340 Mar 21 2014 /etc/cluster/cluster.conf
root@vwsrv3:/etc/pve/nodes#
root@vwsrv3:/etc/pve/nodes# diff /etc/cluster/cluster.conf /etc/pve/cluster.conf
2c2
< <cluster name="VW-Cluster" config_version="10">
---
> <cluster name="VW-Cluster" config_version="9">
root@vwsrv3:/etc/pve/nodes#

Dieter
 
hpcraith said:
34 1 -r--r----- 1 root www-data 339 Mar 21 2014 /etc/pve/cluster.conf
1794903 4 -rw-r--r-- 1 root root 340 Mar 21 2014 /etc/cluster/cluster.conf
root@vwsrv3:/etc/pve/nodes#
root@vwsrv3:/etc/pve/nodes# diff /etc/cluster/cluster.conf /etc/pve/cluster.conf
2c2
< <cluster name="VW-Cluster" config_version="10">
---
> <cluster name="VW-Cluster" config_version="9">
root@vwsrv3:/etc/pve/nodes#

Dieter
Click to expand...

Hi Dieter,
than copy the file and try again!
Code: 
cp /etc/pve/cluster.conf /etc/cluster/cluster.conf
Udo
 
udo said:
Hi Dieter,
than copy the file and try again!
Code: 
cp /etc/pve/cluster.conf /etc/cluster/cluster.conf
Udo
Click to expand...

Your suggestion did not work. I edited vwsrv3's /etc/cluster.conf and changed the version number.
Then suddenly I had the quorum needed. But still I could not change the the xxx.conf I wanted to change
on vwsrv3. I repeated the commands
service cman restart
service pvestatd restart
service pve-cluster restart
several times on my 3 nodes.
Suddenly it worked.
Unfortunately I do not understand why.
I would really appreciate a good description of the cluster filesystem.
Dieter
 
You must log in or register to reply here.
Top Bottom