Proxmox Cluster with Ceph on 4 Nodes - but need the Quorum healthy with 2 nodes

[hrehm]

Hello Board,

working on a proxmox cluster with 4 nodes.
2 Nodes are Servers for VM's containing all sorts of systems services mostly based on some Linux Distris such as Debian/Ubuntu etc. Those 2 Nodes are running HA and Ceph so they are completely redundant to each others.
2 Nodes are SDN Appliances containing VMs for network operating systems. Those VMs are running stuff like VyOS, OcNOS, ROS, vMX etc. They should not be part of Ceph but they should be part of the Cluster (to have common centralized mgmt).

Principally all works, I install networking VMs on local storage and systems VMs on Ceph Storage and that works neatly.

The issue I have is that I have the Quorum going offline when I switch off the TWO networking Appliances (therefore only having 2 out of 4 cluster nodes offline).
Then, Ceph and the VM's using Ceph, goes offline.

Is a way to setup Ceph with 2 Nodes only to have a cluster with 4 Nodes - but only with 2 Nodes required to keep the Quorum active.

Is that setup even possible?

Thanks,
Heiko Rehm

 

[leesteken]

Ceph does not work with two nodes (and no redundancy with three), so this might be a problem for your setup. You should add a QDevice ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_qdevice_technical_overview ) for a fifth vote, to allow for 2 nodes to go down but continuous cluster operation.