unable to modify anything under /etc/pve

learnstuff

New Member
Jul 13, 2012
4
0
1
Platform: Proxmox 2.0

Problem: unable to modify anything under /etc/pve specifically I want to change /etc/pve/qemu-server/103.conf

Error: "error writing /etc/pve/qemu-server/103.conf: Permission Denied"

the file looks like:
-r--r----- 1 root www-data 215 Mar 2 18:30 /etc/pve/qemu-server/103.conf

if I attempt to make it writable:
chmod +w /etc/pve/qemu-server/103.conf
I get:
"chmod: changing permissions of "/etc/pve/qemu-server/103.conf": Function not implemented"

mount says:
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)

What I think the problem is:

The proxmox server with the problem is called node1, I joined it with a second proxmox server (node2) to make a 2 node cluster. Then node2 was deleted without folling the proper procedure to remove a cluster member, so I did not do "pvecm delnode node2" before node2 was permanently powered off.

So, on node1, if I reboot... it says:

"waiting for quorum..." (~30 seconds pass) then "Timed-out waiting for cluster [FAILED]"

once it boots, I login and issue issue "pvecm status" i get:

root@node1:~# pvecm status
Version: 6.2.0
Config Version: 2
Cluster Name: dlabcluster
Cluster Id: 11828
Cluster Member: Yes
Cluster Generation: 192
Membership state: Cluster-Member
Nodes: 1
Expected votes: 2
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
Active subsystems: 1
Flags:
Ports Bound: 0
Node name: node1
Node ID: 1
Multicast addresses: 239.192.46.98
Node addresses: 10.0.0.252
root@node1:~#

The important line I assume is "Quorum: 2 Activity blocked"

To attempt to fix this I then edit /etc/cluster/cluster.conf to remove the dead node2, before editing /etc/cluster/clister.conf it looks like this:

root@node1:~# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="dlabcluster" config_version="2">

<cman keyfile="/var/lib/pve-cluster/corosync.authkey">
</cman>

<clusternodes>
<clusternode name="node1" votes="1" nodeid="1"/>
<clusternode name="node2" votes="1" nodeid="2"/></clusternodes>

</cluster>
root@node1:~#

I edit it to look like:

root@node1:~# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster name="dlabcluster" config_version="2">

<cman keyfile="/var/lib/pve-cluster/corosync.authkey">
</cman>

<clusternodes>
<clusternode name="node1" votes="1" nodeid="1"/>
</clusternodes>

</cluster>
root@node1:~#

and then issue "/etc/init.d/cman restart":

root@node1:~# /etc/init.d/cman restart
Stopping cluster:
Stopping dlm_controld... [ OK ]
Stopping fenced... [ OK ]
Stopping cman... [ OK ]
Waiting for corosync to shutdown:[ OK ]
Unloading kernel modules... [ OK ]
Unmounting configfs... [ OK ]
Starting cluster:
Checking if cluster has been disabled at boot... [ OK ]
Checking Network Manager... [ OK ]
Global setup... [ OK ]
Loading kernel modules... [ OK ]
Mounting configfs... [ OK ]
Starting cman... [ OK ]
Waiting for quorum... [ OK ]
Starting fenced... [ OK ]
Starting dlm_controld... [ OK ]
Unfencing self... [ OK ]
root@node1:~#

and then "pvecm status" shows:

root@node1:~# pvecm status
Version: 6.2.0
Config Version: 2
Cluster Name: dlabcluster
Cluster Id: 11828
Cluster Member: Yes
Cluster Generation: 196
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: node1
Node ID: 1
Multicast addresses: 239.192.46.98
Node addresses: 10.0.0.252
root@node1:~# ~

and now it says "Quorum: 1" which seems good...

but if I try to then edit the qemu config file (the qemu vm is not running)

nano /etc/pve/qemu-server/103.conf

I still get:

Error: "error writing /etc/pve/qemu-server/103.conf: Permission Denied"

I get a similar error if I try to change it via the web interface (says it cant overwrite 103.conf with the new settings temp file)

so I am still unable to change anything mounted under /etc/pve

so I figured I would reboot node1 (since /etc/cluster/cluster.conf was updated) but on reboot it overwrites /etc/cluster/cluster.conf with the copy stored under /etc/pve/cluster.conf that still has node2 listed. The exact boot message is:

"Starting pve cluster filesystem : pve-cluster[dcdb[ notice: wrote new cluster config "/etc/cluster/cluster.conf"
and then it sits at:

"waiting for quorum..." (which fails)

and we are back to "pvecm status" saying

"Quorum: 2 Activity blocked"

I am not sure what to try from here... how do I restore the ability to change files under /etc/pve?

Thanks! -voldial
 
Last edited:
You can't do anything If you loose quorum. So first make sure your cluster setup works and the cluster is quorate.
 
You can't do anything If you loose quorum. So first make sure your cluster setup works and the cluster is quorate.

Hi, thanks for the reply. I understand I need a quorum, that's why I edited /etc/cluster/cluster.conf to remove the dead cluster member and then issued "/etc/init.d/cman restart". After that "pvecm status" says "Quorum: 1" but I am still unable to modify anything under /etc/pve.

I cant bring node2 back up, it's deleted.

pvecm status says I have a quorum...
 
If the other node is already offline you can simply set quorum manually with:

# pvecm expected 1

(set expected votes to 1)
 
Thank you for the reply. I did "pvecm expected 1" but as noted in my first post, "pvecm status" allready shows "Expected Votes; 1" because I edited /etc/cluster/cluster.conf and then did "/etc/init.d/cman restart".

Regardless, executing "pvecm expected 1" still does not make it possible to edit any files mounted under /etc/pve

To recap: I have:
Expected Votes: 1
Total Votes: 1
Node Votes: 1
Quorum: 1

but I am still unable to modify any of the qemu config files, either through the web interface or manually.

How can I manually re-enable write-access to the FUSE filesystem?
 
Ok, fixed. I edited /etc/default/pve-cluster so that DEAMON_OPTS="-l" (local mode) and rebooted. Now I can edit stuff under /etc/pve. I realise this isnt a prefect solution, but that's ok because I am moving the vm's off this box anyway. Thanks again for the neat software.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!