3 nodes mesh cluster using frr (routed with failback)

[jayjayjay]

Hello,

I've recently created a home lab with 3 nodes mesh cluster by following this tutorial (routed with failback using frr): https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#.2Fetc.2Fnetwork.2Finterfaces

It works great:

• I use the mesh network for cluster communication and for ceph communication (cephfs included).
• I have another interface on each node with a linux bridge for VM communication and proxmox nodes access (ssh or WUI)

The main challenge I'm facing now is to give access to the mesh network to some of my VMs as well. The idea would be to give acces to my VM to cephfs + using the mesh network for microk8s cluster communication (which run on 3 dedicated VMs spreaded on the 3 proxmox nodes).

Any would would be much appreciated to understand which config I am supposed to do to reach my goal

Thanks

Jay

 

[vesalius]

You should be ab able to use SDN/vxlan to give VM/LXC access to the mesh network.
https://forum.proxmox.com/threads/i...r-cluster-level-networking.124309/post-543900

 

[jayjayjay]

Hi,

Thanks for you answer but I don't understand how to use Vxlan. I am able to communicate between two VMs in the same VXnet but still unable to communicate wit the node (ping time out).

Thanks

Jay

 

[spirit]

Hi,

Thanks for you answer but I don't understand how to use Vxlan. I am able to communicate between two VMs in the same VXnet but still unable to communicate wit the node (ping time out).

Thanks

Jay

If you want to communicate with the node, you need an ip in the vxlan for the node.
It's not avaiable in the gui, but you can

/etc/network/interfaces

iface <vnet>
     address ....

It'll be merged with the sdn config , setting an ip on the vnet for this node only.

 

[spirit]

HI,

Thanks for your help.

Let me explain a little bit more my config:

• 3 nodes with 3 interfaces:
  • 1 directly connected to my router and used with a linux bridge for VM communication (to the "rest of the world) and proxmox nodes access (ssh or WUI): 192.168.1.0/24
  • 2 interfaces in mesh setup (using RTSP Loop) to connect the 3 nodes together. This subnet is used for cluster communication and for ceph communication (cephfs included): 10.10.10.0/24

So If I understand correctly, I have to create a third subnet (for example 10.10.11.0/24) for VM and Node communication using SDN? Then the ceph mon and mds will still listen only on 10.10.10.0/24 and not on 10.10.11/24, am I missing something here?

The goal is really to give access to cephfs from my VM via the mesh network.

Thanks in advance for the clarification and sorry if my questions looks stupid!

Jay

you need to run your ceph public_network in vxlan too. (10.10.11.0/24)
It's not possible to use 2 différents subnet for ceph, and it's no possible to route between vxlan && real network.

 

[jayjayjay]

you need to run your ceph public_network in vxlan too. (10.10.11.0/24)
It's not possible to use 2 différents subnet for ceph, and it's no possible to route between vxlan && real network.

Thanks for your help. Now everything is clear!

 

[adamli]

Hi jay, I got the use the same way as you to build my pve-ceph cluster and I got several problems:
①how to create pve-cluster? the dialog didn't show my frr network address and lo link.

②when follow the tutorial you mentioned pveceph init --network <node ip> for three nodes, I might got three different cluster here and there weren't connecting for each other.
how do i correctly create that ceph cluster?
I'd be appreciated if you could give some tips!

 

[jayjayjay]

Hi jay, I got the use the same way as you to build my pve-ceph cluster and I got several problems:
①how to create pve-cluster? the dialog didn't show my frr network address and lo link.
View attachment 56325
②when follow the tutorial you mentioned pveceph init --network <node ip> for three nodes, I might got three different cluster here and there weren't connecting for each other.
how do i correctly create that ceph cluster?
I'd be appreciated if you could give some tips!

Hi,
At the end, I decided to use RSTP as it was easier to implement. So sorry but I cannot help you on this.

Cheers

Jay

 

[adamli]

Hi,
At the end, I decided to use RSTP as it was easier to implement. So sorry but I cannot help you on this.

Cheers

Jay

Thanks jay, as you mentioned, the RSTP way in the manual show that there will be a artificial `vmbr1`. It seems that the mesh network are pretend to convert into a normal switch network so it will be more convenient to follow the gui setup? I've reinstall pve on each node, and I'll try this way soon.