unable to modify anything under /etc/pve

Discussion in 'Proxmox VE: Installation and configuration' started by learnstuff, Jul 13, 2012.

  1. learnstuff

    learnstuff New Member

    Joined:
    Jul 13, 2012
    Messages:
    4
    Likes Received:
    0
    Platform: Proxmox 2.0

    Problem: unable to modify anything under /etc/pve specifically I want to change /etc/pve/qemu-server/103.conf

    Error: "error writing /etc/pve/qemu-server/103.conf: Permission Denied"

    the file looks like:
    -r--r----- 1 root www-data 215 Mar 2 18:30 /etc/pve/qemu-server/103.conf

    if I attempt to make it writable:
    chmod +w /etc/pve/qemu-server/103.conf
    I get:
    "chmod: changing permissions of "/etc/pve/qemu-server/103.conf": Function not implemented"

    mount says:
    /dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)

    What I think the problem is:

    The proxmox server with the problem is called node1, I joined it with a second proxmox server (node2) to make a 2 node cluster. Then node2 was deleted without folling the proper procedure to remove a cluster member, so I did not do "pvecm delnode node2" before node2 was permanently powered off.

    So, on node1, if I reboot... it says:

    "waiting for quorum..." (~30 seconds pass) then "Timed-out waiting for cluster [FAILED]"

    once it boots, I login and issue issue "pvecm status" i get:

    root@node1:~# pvecm status
    Version: 6.2.0
    Config Version: 2
    Cluster Name: dlabcluster
    Cluster Id: 11828
    Cluster Member: Yes
    Cluster Generation: 192
    Membership state: Cluster-Member
    Nodes: 1
    Expected votes: 2
    Total votes: 1
    Node votes: 1
    Quorum: 2 Activity blocked
    Active subsystems: 1
    Flags:
    Ports Bound: 0
    Node name: node1
    Node ID: 1
    Multicast addresses: 239.192.46.98
    Node addresses: 10.0.0.252
    root@node1:~#

    The important line I assume is "Quorum: 2 Activity blocked"

    To attempt to fix this I then edit /etc/cluster/cluster.conf to remove the dead node2, before editing /etc/cluster/clister.conf it looks like this:

    root@node1:~# cat /etc/cluster/cluster.conf
    <?xml version="1.0"?>
    <cluster name="dlabcluster" config_version="2">

    <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
    </cman>

    <clusternodes>
    <clusternode name="node1" votes="1" nodeid="1"/>
    <clusternode name="node2" votes="1" nodeid="2"/></clusternodes>

    </cluster>
    root@node1:~#

    I edit it to look like:

    root@node1:~# cat /etc/cluster/cluster.conf
    <?xml version="1.0"?>
    <cluster name="dlabcluster" config_version="2">

    <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
    </cman>

    <clusternodes>
    <clusternode name="node1" votes="1" nodeid="1"/>
    </clusternodes>

    </cluster>
    root@node1:~#

    and then issue "/etc/init.d/cman restart":

    root@node1:~# /etc/init.d/cman restart
    Stopping cluster:
    Stopping dlm_controld... [ OK ]
    Stopping fenced... [ OK ]
    Stopping cman... [ OK ]
    Waiting for corosync to shutdown:[ OK ]
    Unloading kernel modules... [ OK ]
    Unmounting configfs... [ OK ]
    Starting cluster:
    Checking if cluster has been disabled at boot... [ OK ]
    Checking Network Manager... [ OK ]
    Global setup... [ OK ]
    Loading kernel modules... [ OK ]
    Mounting configfs... [ OK ]
    Starting cman... [ OK ]
    Waiting for quorum... [ OK ]
    Starting fenced... [ OK ]
    Starting dlm_controld... [ OK ]
    Unfencing self... [ OK ]
    root@node1:~#

    and then "pvecm status" shows:

    root@node1:~# pvecm status
    Version: 6.2.0
    Config Version: 2
    Cluster Name: dlabcluster
    Cluster Id: 11828
    Cluster Member: Yes
    Cluster Generation: 196
    Membership state: Cluster-Member
    Nodes: 1
    Expected votes: 1
    Total votes: 1
    Node votes: 1
    Quorum: 1
    Active subsystems: 5
    Flags:
    Ports Bound: 0
    Node name: node1
    Node ID: 1
    Multicast addresses: 239.192.46.98
    Node addresses: 10.0.0.252
    root@node1:~# ~

    and now it says "Quorum: 1" which seems good...

    but if I try to then edit the qemu config file (the qemu vm is not running)

    nano /etc/pve/qemu-server/103.conf

    I still get:

    Error: "error writing /etc/pve/qemu-server/103.conf: Permission Denied"

    I get a similar error if I try to change it via the web interface (says it cant overwrite 103.conf with the new settings temp file)

    so I am still unable to change anything mounted under /etc/pve

    so I figured I would reboot node1 (since /etc/cluster/cluster.conf was updated) but on reboot it overwrites /etc/cluster/cluster.conf with the copy stored under /etc/pve/cluster.conf that still has node2 listed. The exact boot message is:

    "Starting pve cluster filesystem : pve-cluster[dcdb[ notice: wrote new cluster config "/etc/cluster/cluster.conf"
    and then it sits at:

    "waiting for quorum..." (which fails)

    and we are back to "pvecm status" saying

    "Quorum: 2 Activity blocked"

    I am not sure what to try from here... how do I restore the ability to change files under /etc/pve?

    Thanks! -voldial
     
    #1 learnstuff, Jul 13, 2012
    Last edited: Jul 13, 2012
  2. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,375
    Likes Received:
    291
    You can't do anything If you loose quorum. So first make sure your cluster setup works and the cluster is quorate.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. learnstuff

    learnstuff New Member

    Joined:
    Jul 13, 2012
    Messages:
    4
    Likes Received:
    0
    Hi, thanks for the reply. I understand I need a quorum, that's why I edited /etc/cluster/cluster.conf to remove the dead cluster member and then issued "/etc/init.d/cman restart". After that "pvecm status" says "Quorum: 1" but I am still unable to modify anything under /etc/pve.

    I cant bring node2 back up, it's deleted.

    pvecm status says I have a quorum...
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,375
    Likes Received:
    291
    If the other node is already offline you can simply set quorum manually with:

    # pvecm expected 1

    (set expected votes to 1)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. learnstuff

    learnstuff New Member

    Joined:
    Jul 13, 2012
    Messages:
    4
    Likes Received:
    0
    Thank you for the reply. I did "pvecm expected 1" but as noted in my first post, "pvecm status" allready shows "Expected Votes; 1" because I edited /etc/cluster/cluster.conf and then did "/etc/init.d/cman restart".

    Regardless, executing "pvecm expected 1" still does not make it possible to edit any files mounted under /etc/pve

    To recap: I have:
    Expected Votes: 1
    Total Votes: 1
    Node Votes: 1
    Quorum: 1

    but I am still unable to modify any of the qemu config files, either through the web interface or manually.

    How can I manually re-enable write-access to the FUSE filesystem?
     
  6. learnstuff

    learnstuff New Member

    Joined:
    Jul 13, 2012
    Messages:
    4
    Likes Received:
    0
    Ok, fixed. I edited /etc/default/pve-cluster so that DEAMON_OPTS="-l" (local mode) and rebooted. Now I can edit stuff under /etc/pve. I realise this isnt a prefect solution, but that's ok because I am moving the vm's off this box anyway. Thanks again for the neat software.
     
  7. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,375
    Likes Received:
    291
    You need to restart the fs after restarting cman:

    # /etc/init.d/pve-cluster restart
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice