[SOLVED] Proxmox VE cluster node shows unknown

Discussion in 'Proxmox VE: Installation and configuration' started by Hivane, Dec 4, 2018.

  1. Hivane

    Hivane Member

    Joined:
    Feb 11, 2014
    Messages:
    38
    Likes Received:
    0
    Hello all,

    I am having an issue within a 3 nodes cluster:
    After a disk failure, I have re installed one of the nodes (using the same hostname/IP).
    Before trying to make it join the cluster again, I removed it from the cluster using "pvecm delnode", but something went bad. I then, tried to remove it again, re-reinstall it using a different hostname, and add it to the cluster.

    Now, the cluster has the three nodes, pvecm status shows a normal 3-nodes output, but I have issues when I log on the web interface: Only the node I connect to, appears as green, and all the others appears as unknown.

    You can have a view of pvecm status & how it appears on the webinterface here:
    https://twitter.com/acontios_net/status/1069844561944567808
    and here:
    https://twitter.com/acontios_net/status/1069852107057098752

    On the first tweet, I am logged in on vdg-pve01-par6 webinterface, and on the second one, on vdg-pvefiler webinterface. All the VMs on pve01 are working fine.

    How could I remove myself from that stucked situation ?

    Thanks for your help !

    Clément
     
  2. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    1,269
    Likes Received:
    117
    Please check the output of `systemctl status -l pvestatd` - and restart the service, also check your journal `journalctl -r`
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Hivane

    Hivane Member

    Joined:
    Feb 11, 2014
    Messages:
    38
    Likes Received:
    0
    Hi Stoiko,

    it didnt unfortunately solve any problem to restart it (on each of the nodes)
    Journal does not show anything relevant
     
  4. bizzarrone

    bizzarrone Member

    Joined:
    Nov 27, 2014
    Messages:
    42
    Likes Received:
    1
    Good morning.
    Just installed proxmox 5.3, updated just with Debian updates. No pve-nosubs repository. I created a a 3 nodes cluster.
    I am using a separate network for the cluster 10 GB. hosts file correct.
    Immediatly after creation of the cluster:

    Code:
    Quorum information
    ------------------
    Date:             Wed Dec  5 10:15:03 2018
    Quorum provider:  corosync_votequorum
    Nodes:            3
    Node ID:          0x00000001
    Ring ID:          1/376
    Quorate:          Yes
    
    Votequorum information
    ----------------------
    Expected votes:   3
    Highest expected: 3
    Total votes:      3
    Quorum:           2 
    Flags:            Quorate
    
    Membership information
    ----------------------
        Nodeid      Votes Name
    0x00000001          1 10.1.1.1 (local)
    0x00000002          1 10.1.1.2
    0x00000003          1 10.1.1.3
    
    only 2 minutes after...

    Code:
    Quorum information
    ------------------
    Date:             Wed Dec  5 10:03:15 2018
    Quorum provider:  corosync_votequorum
    Nodes:            1
    Node ID:          0x00000001
    Ring ID:          1/16
    Quorate:          No
    
    Votequorum information
    ----------------------
    Expected votes:   3
    Highest expected: 3
    Total votes:      1
    Quorum:           2 Activity blocked
    Flags:
    
    then I restarted corosync
    Code:
    systemctl restart corosync
    
    then it turns ok.. after 2 minutes, the same error ...
     
  5. bizzarrone

    bizzarrone Member

    Joined:
    Nov 27, 2014
    Messages:
    42
    Likes Received:
    1
    Good morning again.
    I discovered the multicast traffic is blocked after 2 minutes...
    Code:
    prox03 : multicast, seq=180, size=69 bytes, dist=0, time=0.385ms
    prox02 : multicast, seq=180, size=69 bytes, dist=0, time=0.421ms
    prox03 :   unicast, seq=181, size=69 bytes, dist=0, time=0.222ms
    prox02 :   unicast, seq=181, size=69 bytes, dist=0, time=0.360ms
    prox03 : multicast, seq=181, size=69 bytes, dist=0, time=0.394ms
    prox02 : multicast, seq=181, size=69 bytes, dist=0, time=0.436ms
    prox02 :   unicast, seq=182, size=69 bytes, dist=0, time=0.403ms
    prox03 :   unicast, seq=182, size=69 bytes, dist=0, time=0.384ms
    prox03 :   unicast, seq=183, size=69 bytes, dist=0, time=0.214ms
    prox02 :   unicast, seq=183, size=69 bytes, dist=0, time=0.345ms
    
    then

    Code:
    prox02 :   unicast, xmt/rcv/%loss = 378/378/0%, min/avg/max/std-dev = 0.151/0.390/0.604/0.043
    prox02 : multicast, xmt/rcv/%loss = 378/180/52% (seq>=2 52%), min/avg/max/std-dev = 0.277/0.437/0.594/0.035
    prox03 :   unicast, xmt/rcv/%loss = 378/378/0%, min/avg/max/std-dev = 0.188/0.342/0.491/0.065
    prox03 : multicast, xmt/rcv/%loss = 378/180/52% (seq>=2 52%), min/avg/max/std-dev = 0.311/0.400/0.492/0.028
    
    so I am turning into unicast traffic...
     
  6. Hivane

    Hivane Member

    Joined:
    Feb 11, 2014
    Messages:
    38
    Likes Received:
    0
    Solved the problem with the following step on each node::

    Code:
    killall corosync -9 
    umount /etc/pve -l 
    service pve-cluster stop 
    
    ... wait a bit, re-installed the same pve-cluster package on each node, and re started everything.
     
  7. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    1,269
    Likes Received:
    117
    Glad (both) Problems were resolved ! Please mark the thread as solved :)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Hivane

    Hivane Member

    Joined:
    Feb 11, 2014
    Messages:
    38
    Likes Received:
    0
    Done! :)
     
  9. bizzarrone

    bizzarrone Member

    Joined:
    Nov 27, 2014
    Messages:
    42
    Likes Received:
    1
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice