Cluster und die Nodes sehen sich nicht mehr

Discussion in 'Proxmox VE (Deutsch)' started by josch12, Aug 10, 2018.

  1. josch12

    josch12 New Member

    Joined:
    Oct 20, 2017
    Messages:
    18
    Likes Received:
    0
    Hallo,

    gestern Abend ist aufgefallen das unser Cluster mit 7 Nodes krank ist.

    wenn ich mich z.b auf Node 1 einlogge, sehe ich die Vms auf dem Node 1, die der anderen Nodes allerdings nicht.

    [​IMG]

    gehe ich per SSH auf einen Node, so kann ich die anderen pingen, bzw komme mittels SSH ohne Probleme drauf.

    Ich sehe über die Gui auch die Auslastung der anderen Nodes

    [​IMG]

    Der Status Vom Cluster ist ebenfalls normal, außer das ich keine Werte erhalte.

    [​IMG]

    Jemand eine Idee wo ich ansetzen könnte ?

    am Cluster selbst wurde gestern nichts geändert, nur an einer VPN Verbindung vom RZ in unser Büro, hängt allerdings nicht damit zusammen ( sind schon einmal andere Netze )
     
    #1 josch12, Aug 10, 2018
  2. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,917
    Likes Received:
    266
    steht im syslog irgendwas?

    wenn der pvestatd irgendwie hängt können diese symptome auftreten, ist jetzt nur die frage warum der hängt
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #2 dcsapak, Aug 10, 2018
  3. josch12

    josch12 New Member

    Joined:
    Oct 20, 2017
    Messages:
    18
    Likes Received:
    0
    Hallo,

    guter Ansatz ...

    ich habe dort einige corosync Alerts gefunden

    Aug 10 06:25:08 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:08 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:07 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:07 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:07 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:07 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:07 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:07 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:08 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:08 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:10 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:10 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:10 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:10 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:10 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:10 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:11 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:12 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 pveproxy[14053]: worker exit
    Aug 10 06:25:13 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:13 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: error [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: alert [TOTEM ] Invalid packet data
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Digest does not match
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Received message has invalid digest... ignoring.
    Aug 10 06:25:14 prox3 corosync[2040]: [TOTEM ] Invalid packet data
     
    #3 josch12, Aug 10, 2018
  4. Markus Thormann

    Markus Thormann Member

    Joined:
    Oct 25, 2015
    Messages:
    291
    Likes Received:
    13
    Komplett durch Booten alle dann gehts wieder. Liegt ggf gerade an der Version oder Kernel.
     
    #4 Markus Thormann, Aug 10, 2018
  5. josch12

    josch12 New Member

    Joined:
    Oct 20, 2017
    Messages:
    18
    Likes Received:
    0
    Dankeschön ! , hat funktioniert.
     
    #5 josch12, Aug 10, 2018
  6. Markus Thormann

    Markus Thormann Member

    Joined:
    Oct 25, 2015
    Messages:
    291
    Likes Received:
    13
    Hoffe das wird in einem Update gefixt. Aber ich darf ja nichts sagen das Produkt ist ja zu 100% zuverlässig.
     
    #6 Markus Thormann, Aug 11, 2018
  7. fireon

    fireon Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Oct 25, 2010
    Messages:
    2,690
    Likes Received:
    148
    Ist aber keine Lösung. Solche Probleme sollten/müssen immer aufgearbeitet werden. Fast immer liegt es an einem Netzwerk oder HW Problem. Wir haben seit Jahren einige Cluster am laufen. Bis jetzt waren es immer defekte Netzwerkkarten, Switches und Kabeln. Einmal hatten wir ein Netzwerkkartentreiberproblem mit einer Netxen Karte. Das war damals ein Kernelproblem.

    Welche Netzwerkkarten hast du verbaut? Siehst du zu diesem Zeitpunkt in dmesg was brauchbares?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #7 fireon, Aug 12, 2018
  8. Markus Thormann

    Markus Thormann Member

    Joined:
    Oct 25, 2015
    Messages:
    291
    Likes Received:
    13
    Kann auch an der Software liegen so wieindem Fall es ggf. So ist. Wir haben auch seit Jahren ein Cluster laufen und durch fehlerhafte Updates in der Software gibt es einfach mehr Probleme solcher Art.
     
    #8 Markus Thormann, Aug 12, 2018
(You must log in or sign up to post here.)
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Accept Learn More...
    Dismiss Notice