Removing node from cluster does not remove it from gui.

Discussion in 'Proxmox VE: Installation and configuration' started by TobiasE, Apr 4, 2018.

  1. TobiasE

    TobiasE New Member

    Joined:
    Dec 6, 2017
    Messages:
    24
    Likes Received:
    0
    Hello
    After removing a node from my proxmoxcluster, it still shows up in my cluster with a white x on it.
    I tried restarting corosync and pveproxy as well as the filesystem.
    I see that there is still an entry for the deleted node in /etc/pve/corosync,conf, however I am unable to edit this file (readonly). Is this a known bug?
    I used
    pvecm delnode <node> to delete the node to delete it
     
    #1 TobiasE, Apr 4, 2018
    Last edited: Apr 4, 2018
  2. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,138
    Likes Received:
    148
    Hi,

    No known bug, the removal had not completed correctly though.

    Do you had a 2 node setup where you removed one?

    Can you run:
    Code:
    pvecm expected 1
    
    To tell the other node it's fine with just one node, and then rerun the 'pvecm delnode NODE' command?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. TobiasE

    TobiasE New Member

    Joined:
    Dec 6, 2017
    Messages:
    24
    Likes Received:
    0
    I had four nodes (now I have three)
     
  4. TobiasE

    TobiasE New Member

    Joined:
    Dec 6, 2017
    Messages:
    24
    Likes Received:
    0
    To clarify the removed node appears in the web gui only.
     
  5. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,138
    Likes Received:
    148
    where all (now remaining) three online and fully operable when doing the deletion?
    can you post:
    Code:
    pvecm status
    
    Can you please check if the following situation applies to you:
    https://forum.proxmox.com/threads/cluster-node-stuck-in-ui.42330/#post-203778
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. TobiasE

    TobiasE New Member

    Joined:
    Dec 6, 2017
    Messages:
    24
    Likes Received:
    0
    I have removed the nodedirectory
     
  7. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,138
    Likes Received:
    148
    So all well now?

    Just a bit concerned because the node shouldn't appear in the corosync.conf anymore after the 'pvecm delnode NODE'...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. TobiasE

    TobiasE New Member

    Joined:
    Dec 6, 2017
    Messages:
    24
    Likes Received:
    0
    No it does not appear when i run pvecm nodes on a node which is still in the cluster, but in the web gui it still appears.
     
  9. uzer

    uzer New Member

    Joined:
    Dec 19, 2018
    Messages:
    4
    Likes Received:
    0
    I am experiencing the same issue on PVE 5.3-5.
    I've manually decremented number of expected nodes (since it didn't automatically decremented when I removed a node from the cluster), but the GUI 'still remembers' the deleted node.

    Code:
    pvecm status
    
    Quorum information
    ------------------
    Date:             Wed Dec 19 12:03:01 2018
    Quorum provider:  corosync_votequorum
    Nodes:            4
    Node ID:          0x00000002
    Ring ID:          2/72
    Quorate:          Yes
    
    Votequorum information
    ----------------------
    Expected votes:   4
    Highest expected: 4
    Total votes:      4
    Quorum:           3 
    Flags:            Quorate 
    
    Membership information
    ----------------------
        Nodeid      Votes Name
    0x00000002          1 172.2.2.3 (local)
    0x00000005          1 172.2.2.6
    0x00000001          1 172.2.2.2
    0x00000003          1 172.2.2.4
    
     
  10. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,138
    Likes Received:
    148
    that probably means that the removed node is still online and still means it's in the cluster, I'd guess.
    Does the deleted node still has configuration files in it's /etc/pve/nodes/NODENAME/ lxc and qemu-server folders? Because if, then the web gui creates a node in the resource tree no matter what, as it needs to map VM and CT configs somewhere. If only that's the case delete them and reload.

    how did you do this exactly? Did you follow:
    https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_remove_a_cluster_node
    or a section below, without purging the removed node:
    https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_separate_node_without_reinstall
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. uzer

    uzer New Member

    Joined:
    Dec 19, 2018
    Messages:
    4
    Likes Received:
    0
    Yes, the node was still online (I thought stopping pve services would be enough).
    Just ifdown'ed LAN interface (the nodes were addressing each other by LAN IPs), the issue is still there.

    No, those dirs are empty on the removed node and there's no /etc/pve/nodes/REMOVEDNODENAME/ on the online nodes of the cluster.

    I probably did all those actions in a wrong order: I removed the node via `pvecm delnode` BEFORE I switched off the removed node's network.
    How do I unscrew things now?
     
  12. uzer

    uzer New Member

    Joined:
    Dec 19, 2018
    Messages:
    4
    Likes Received:
    0
    Oh, god, I screwed up big time:
    I'm experiencing an issue like the one described here: https://forum.proxmox.com/threads/gui-node-list-empty-during-vm-creation-migration.42739/
    and `pvesh get /nodes` returns `unable to read '/etc/pve/nodes/d0217/pve-ssl.pem' - No such file or directory`, where d0217 is the name of the removed node.
    Afair, I removed that dir manually, `pvecm status` and `pvecm nodes` don't list d0217 anymore.
    What do I do?
     
  13. uzer

    uzer New Member

    Joined:
    Dec 19, 2018
    Messages:
    4
    Likes Received:
    0
    FTH: the issue got resolved via `sudo systemctl restart corosync && sudo systemctl restart pve-cluster` on each alive node of the cluster.
    Thank you!
     
  14. hans05

    hans05 New Member

    Joined:
    Mar 4, 2019
    Messages:
    6
    Likes Received:
    0
    Just for information.

    I had the same problem and in addition to all of the above I had to edit manually the /etc/pve/corosync.conf file.
    I removed the node-entry and increased the value of config_version before restarting corosync and pve-cluster.
    After all of that everything worked fine.

    BUT, this is not the expected behaviour since I sticked absolutely to the officially described way of removing the node!
    Mhhh....
     
  15. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,138
    Likes Received:
    148
    It surely isn't we try to remove it from the configuration file actively, if anything is off with the cluster (e.g., not quorate without the "to-be-deleted" node) then this may fail.
    The 'pvecm delnode' command should show errors then, though.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice