Hi,
I'm planning to create a full mesh Thunderbolt 4 network between my nodes (actually it is a 3 nodes cluster so the full mesh is a ring). The plan is to use it as the private and cluster networks for Ceph instead of 2.5 Gbit NICs I'm currently using for obvious performances reasons (and isolation too).
I think I'm quite confortable with setting all this up using an OpenFabric SDN fabric, but then I have an additional requirement: Each node is hosting a kubernetes node virtual machine, and I'm using the Ceph CSI as the main storage for the K8s cluster. So it means that those VMs need to have connectivity to the Ceph network too.
And here I admit I'm a bit lost. I could create a bridge on each physical node, connect a veth to that bridge, and route from this bridge to the Ceph network. However it means that each VM "ceph" veth must be in its own subnet and then it becomes impossible to migrate those VM between physical nodes. I would also need to manually add static routes to the ceph network in each VM.
Migrating is in that case not critical because the k8s should handle failing nodes, but if someone has a better way to setup this and is ok to share the solution it would be very nice.
Thank you.
I'm planning to create a full mesh Thunderbolt 4 network between my nodes (actually it is a 3 nodes cluster so the full mesh is a ring). The plan is to use it as the private and cluster networks for Ceph instead of 2.5 Gbit NICs I'm currently using for obvious performances reasons (and isolation too).
I think I'm quite confortable with setting all this up using an OpenFabric SDN fabric, but then I have an additional requirement: Each node is hosting a kubernetes node virtual machine, and I'm using the Ceph CSI as the main storage for the K8s cluster. So it means that those VMs need to have connectivity to the Ceph network too.
And here I admit I'm a bit lost. I could create a bridge on each physical node, connect a veth to that bridge, and route from this bridge to the Ceph network. However it means that each VM "ceph" veth must be in its own subnet and then it becomes impossible to migrate those VM between physical nodes. I would also need to manually add static routes to the ceph network in each VM.
Migrating is in that case not critical because the k8s should handle failing nodes, but if someone has a better way to setup this and is ok to share the solution it would be very nice.
Thank you.