pvecm delnode didn't worked

[Ivan Cacija]

Hello guys,

I need help with my cluster.

I have tried to add a node, but there is some issue with network at the time of adding node and node hasn't been fully added to cluster, so tried to remove it.

As soon as I tried to add it to cluster, cluster just gone mad, and I can connect to single node in cluster (GUI), but I can't manage anything from GUI.

Firstly I have run:

pvecm delnode pve-xyz11

Error I got is:

error with cfs lock 'file-corosync_conf': got lock request timeout

when I run pvecm status I got:

pvecm status
Quorum information
------------------
Date:             Sun Dec 29 01:20:56 2019
Quorum provider:  corosync_votequorum
Nodes:            5
Node ID:          0x00000001
Ring ID:          1.40
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   6
Highest expected: 6
Total votes:      5
Quorum:           4
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 ip1 (local)
0x00000002          1 ip2
0x00000003          1 ip3
0x00000004          1 ip4
0x00000005          1 ip5

corosync.conf looks like this:

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve-xyz
    nodeid: 1
    quorum_votes: 1
    ring0_addr: ip1
  }
  node {
    name: pve-xyz11
    nodeid: 6
    quorum_votes: 1
    ring0_addr: ip2
  }
  node {
    name: pve-xyz2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: ip3
  }
  node {
    name: pve-xyz3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: ip4
  }
  node {
    name: pve-xyz4
    nodeid: 4
    quorum_votes: 1
    ring0_addr: ip5
  }
  node {
    name: pve-xyz5
    nodeid: 5
    quorum_votes: 1
    ring0_addr: ip6
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: XYZ1
  config_version: 6
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  secauth: on
  version: 2
}

and /etc/pve/.members

{
"nodename": "pve-xyz",
"version": 12,
"cluster": { "name": "XYZ1", "version": 6, "nodes": 6, "quorate": 1 },
"nodelist": {
  "pve-xyz5": { "id": 5, "online": 1, "ip": "ip1"},
  "pve-xyz": { "id": 1, "online": 1, "ip": "ip2"},
  "pve-xyz2": { "id": 2, "online": 1, "ip": "ip3"},
  "pve-xyz3": { "id": 3, "online": 1, "ip": "ip4"},
  "pve-xyz4": { "id": 4, "online": 1, "ip": "ip5"},
  "pve-xyz11": { "id": 6, "online": 0}
  }
}

So, how can I del this pve-xyz11 and run normal cluster again.

Thank you,

Ivan

 

[wolfgang]

Hi,

I guess there is a service hanging.
Restart the following services on all nodes.

systemctl restart pvedaemon pveproxy pvestatd pve-cluster