Hi
Yesterday we took a server in our PVE cluster out of production.
As per the documentation we can "pvecm delnode <nodename>" to remove the node from the cluster.
Immediately after removing the node, it was no longer visible from "pvecm status" and "pvecm nodes", but we could still see it in the Web GUI on the list of servers in the datacenter view.
I then searched the forums to see if others have had the same issue and found that I could delete /etc/pve/nodes/<nodename>. I did that, but nothing changed.
I tried to restart pvedaemon, pveproxy and pve-cluster on all nodes. Nothing changed on the remaining nodes.
However on the removed node, we do now get "Cannot initialize CMAP service" when running "pvecm status" or "pvecm nodes".
I did also try to reboot the removed node, but this only made it show up as offline and as soon as it was online again, we could see that it was online.
How can we get the node out of the system?
pveversion -V of the removed node:
pveversion -V of an existing running cluster node:
I do not know if this is of any use, but we are running with OVH dedicated servers and their vRack system, which means that the cluser is running in unicast-mode.
Yesterday we took a server in our PVE cluster out of production.
As per the documentation we can "pvecm delnode <nodename>" to remove the node from the cluster.
Immediately after removing the node, it was no longer visible from "pvecm status" and "pvecm nodes", but we could still see it in the Web GUI on the list of servers in the datacenter view.
I then searched the forums to see if others have had the same issue and found that I could delete /etc/pve/nodes/<nodename>. I did that, but nothing changed.
I tried to restart pvedaemon, pveproxy and pve-cluster on all nodes. Nothing changed on the remaining nodes.
However on the removed node, we do now get "Cannot initialize CMAP service" when running "pvecm status" or "pvecm nodes".
I did also try to reboot the removed node, but this only made it show up as offline and as soon as it was online again, we could see that it was online.
How can we get the node out of the system?
pveversion -V of the removed node:
Code:
proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-2 (running version: 5.2-2/b1d1c7f4)
pve-kernel-4.15: 5.2-3
pve-kernel-4.13: 5.1-45
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-3-pve: 4.13.16-49
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.4-1-pve: 4.13.4-26
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-33
libpve-guest-common-perl: 2.0-16
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-28
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9pveversion -V of an existing running cluster node:
Code:
proxmox-ve: 5.2-2 (running kernel: 4.13.16-2-pve)
pve-manager: 5.2-2 (running version: 5.2-2/b1d1c7f4)
pve-kernel-4.15: 5.2-3
pve-kernel-4.13: 5.1-45
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.16-3-pve: 4.13.16-49
pve-kernel-4.13.16-2-pve: 4.13.16-48
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-33
libpve-guest-common-perl: 2.0-16
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-28
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9I do not know if this is of any use, but we are running with OVH dedicated servers and their vRack system, which means that the cluser is running in unicast-mode.