Like many of you have a thunderbolt mesh network that uses openfabric frr to create a routed topology. I use IPv6 for ceph on that mesh along with IPv4 for other traffic (like pve cluster traffic).
I wanted my docker swarm VMs to be able to access the cephFS file store. I am using virtioFS but it has some drawbacks (hookscripts need to check cephFS is actually there at boot), issues if QEMU barfs on the passthrough, and no snapshots allowed of the VM.
I saw a few solutions using SDN and EVPN - the issue for, doesn't work in a dual stack and environment and boy are the tutorials complex.
So i came up with another routing approach that IMO is far simpler that SDN and EVPN *and* supports IPv4 and IPv6.
The basic steps are:
When SDN fully supports IPv6 and IPv6 IPAM i will revist using SDN.
Oh as a bonus link, here is how i enabled LAN clients to do access the mesh using BGP.
lan-access-to-mesh.md
Hope someone finds this useful.
I wanted my docker swarm VMs to be able to access the cephFS file store. I am using virtioFS but it has some drawbacks (hookscripts need to check cephFS is actually there at boot), issues if QEMU barfs on the passthrough, and no snapshots allowed of the VM.
I saw a few solutions using SDN and EVPN - the issue for, doesn't work in a dual stack and environment and boy are the tutorials complex.
So i came up with another routing approach that IMO is far simpler that SDN and EVPN *and* supports IPv4 and IPv6.
The basic steps are:
- have a working mesh network with openfabric
- create a new bridge with no ports dedicated for VMs to access ceph mesh
- add bridge to openfarbric
- configure VM to operate on the new bridge subnet
When SDN fully supports IPv6 and IPv6 IPAM i will revist using SDN.
Oh as a bonus link, here is how i enabled LAN clients to do access the mesh using BGP.
lan-access-to-mesh.md
Hope someone finds this useful.
Last edited:
