Hi
We recently deployed a small 3-node PVE cluster on-premise. All three nodes run Ceph OSD, Ceph Monitor, Ceph Manager and Ceph MDS.
We run the setup with 3 switches and each server has 4 NICs.
The two first NICs are configured as a linux bond in failover mode and have a cable to each of the first two switches. PVE cluster runs here as well
The last two NICs run a linux bond with LACP to the third switch so that we can do maintenance on the primary network without having Ceph loose conectivity. Ceph is the only service running here.
It has been working fine, but a technician misconfigured the switch used for Ceph, so that it could no longer be managed and had to be reset.
In order to fix that, we planned to power off all 3 nodes, reconfigure the switch and then power the nodes on again. Since we had to move things around in the rack, we did that and then the technician forgot to reconfigure the switch and then powered on the servers on again. Since the switch still had the ports configured for LACP, everything still worked.
The technician then remembered that he had to reset the switch and did that while the cluster was online and had virtual machines running. Since LACP is no longer working, we shut the servers down again, completed reconfiguration of the switch, powered on the servers and found that the technician had not configured the switch ports for LACP. Reconfigured the ports while servers were running.
From the point where the switch was reset, "pveceph status" reports "got timeout" and so does the web UI. Running "ceph -s" seems to hang forever, but after some time it outputs this:
We have tried rebooting all servers at the same time, rebooting them one-by-one and wait until the node was back up before rebooting the next. We have also tried "systemctl restart ceph.target" with also no luck.
Looking at the syslog, it seems there is a connectivity issue where all nodes report that they cannot connect to the socket on port 6789 on every other node. Nodes can ping each other on the IP-adresses assigned to Ceph public network. Also we can do a telnet connection to each node on port 6789. So connectiivty should work.
Each node does also report syslog errors for its own IP on port 6789
Has the above mistake broken Ceph and is there a way to have PVE fix the cluster since it is a PVE managed cluster?
We recently deployed a small 3-node PVE cluster on-premise. All three nodes run Ceph OSD, Ceph Monitor, Ceph Manager and Ceph MDS.
We run the setup with 3 switches and each server has 4 NICs.
The two first NICs are configured as a linux bond in failover mode and have a cable to each of the first two switches. PVE cluster runs here as well
The last two NICs run a linux bond with LACP to the third switch so that we can do maintenance on the primary network without having Ceph loose conectivity. Ceph is the only service running here.
It has been working fine, but a technician misconfigured the switch used for Ceph, so that it could no longer be managed and had to be reset.
In order to fix that, we planned to power off all 3 nodes, reconfigure the switch and then power the nodes on again. Since we had to move things around in the rack, we did that and then the technician forgot to reconfigure the switch and then powered on the servers on again. Since the switch still had the ports configured for LACP, everything still worked.
The technician then remembered that he had to reset the switch and did that while the cluster was online and had virtual machines running. Since LACP is no longer working, we shut the servers down again, completed reconfiguration of the switch, powered on the servers and found that the technician had not configured the switch ports for LACP. Reconfigured the ports while servers were running.
From the point where the switch was reset, "pveceph status" reports "got timeout" and so does the web UI. Running "ceph -s" seems to hang forever, but after some time it outputs this:
Code:
2022-07-15T08:29:46.358+0200 7f7367611700 0 monclient(hunting): authenticate timed out after 300Looking at the syslog, it seems there is a connectivity issue where all nodes report that they cannot connect to the socket on port 6789 on every other node. Nodes can ping each other on the IP-adresses assigned to Ceph public network. Also we can do a telnet connection to each node on port 6789. So connectiivty should work.
Each node does also report syslog errors for its own IP on port 6789
Has the above mistake broken Ceph and is there a way to have PVE fix the cluster since it is a PVE managed cluster?
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.39-1-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
Last edited:
