nodes inexplicably shut down in CEPH cluster in proxmox ve 8.1.4

[Jalvarez]

The nodes inexplicably turn off in CEPH cluster in proxmox ve 8.1.4, this happens occasionally when a backup is made of a VM whose storage is in a pool, suddenly the node is inhibited and I have to restart it to make it work again.
I have 3 identical nodes.

Please could someone help me with this problem?

 

[VictorSTS]

Do you use HA?
How many corosync links do you have (corosync-cfgtool -s on each node)?
Which network is used for Ceph public and cluster traffic?
"The nodes inexplicably turn off" means they reboot or they get fully off?

Please elaborate the exact behavior that you are seeing.

 

[Jalvarez]

It's a disaster, everything remains inhibited and cannot be used until the reset button is pressed.

 

[Jalvarez]

This error appears when I migrate the virtual hard drives to the storage created in a ceph pool

 

[Jalvarez]

 

[VictorSTS]

There's little anyone can do to help you unless you provide at least the data I requested:

Do you use HA?
How many corosync links do you have (corosync-cfgtool -s on each node)?
Which network is used for Ceph public and cluster traffic?
"The nodes inexplicably turn off" means they reboot or they get fully off?

 

[jlauro]

CEPH is very demanding on the network with massive network traffic multiplication, and from what I understand, corosync is very sensitive to network blips. What is the networking on the nodes? 1gb? 10g, 100gb? How many nics/node? It does sound like ceph could be saturating the network bandwidth and causing corosync to fail.

 

[Jalvarez]

 

[Jalvarez]

I have three network cards with one connection for the cluster network, another for the administrative network and the other for the VMs at the bridge level, the networks are:

ADMIN VLAN: 172.16.20.0/26
CLUSTER:192.168.60.0/28
VIRTUAL VM: 172.16.20.0/26

 

[jlauro]

If ceph is also on the cluster network that corosync is using (sounds like it is), that is probably saturating the network and disrupting corosync.
I suggest making your ADMIN VLAN (or a different NIC) as your primary link for corosync and leaving the cluster as redundant.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

 

[Jalvarez]

Ok, I have already done it, I am monitoring to see how the cluster behaves with this change in corosync

 

[Jalvarez]

 

[Jalvarez]

The same problems continue to occur, however I have noticed that when I group more than 5 VMs that have CEPH storage the node hangs

 

[Jalvarez]

 

[Jalvarez]

root@172.16.20.10: Permission denied (publickey,password).
TASK ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=srvprox1' root@172.16.20.10 pvecm mtunnel -migration_network 192.168.60.10/28 -get_migration_ip' failed: exit code 255

 

[Jalvarez]

How can I perform migrations within the cluster gives me this error

root@172.16.20.10: Permission denied (publickey,password).
TASK ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=srvprox1' root@172.16.20.10 pvecm mtunnel -migration_network 192.168.60.10/28 -get_migration_ip' failed: exit code 255

 

[jorel83]

How can I perform migrations within the cluster gives me this error

root@172.16.20.10: Permission denied (publickey,password).
TASK ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=srvprox1' root@172.16.20.10 pvecm mtunnel -migration_network 192.168.60.10/28 -get_migration_ip' failed: exit code 255

I saw similar issues on my 8.1 fresh install and could not get any resolution to this problem and still 3 of 4 servers is down after this shit-show. Did you find a solution?