3 servers, 3 cables, 1 ceph network?

[thoth]

1. Can you take 3 servers with 4 port networking and connect them in this fashion:
A - B
A - C
B - C
with 3 cables and have redundant networking? (literal redundant would require 6 cables and eat all ports I think)
2. Are there three nets in the end? (and then possibly connect them all together with wireguard, kind of like you can do with flannel in a kubernetes cluster)
3. Can I get proxmox to put its ceph cluster on this to alleviate storage traffic going through the switch with the rest of the traffic?
4. Is this a terrible idea? All thoughts are welcome.

 

[BD-Nets]

Yes. See here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

 

[czechsys]

Depends on nic utilization, but we use the switch way for our 3 nodes cluster:
1x lacp (2 ports) with vlans for management (= ceph public), for vm, for corosync, etc
2x lacp (2 ports) with vlans for corosync, for ceph storage

Mesh is for small number of nodes or when the switch is too costly for better speeded nics. It's limited to 1 link between nodes.