3 node mesh network VLAn-able?

[CadilLACi]

Hello,

I have 3 servers with dual 10Gbit cards, but no 10g switch.

I read the https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server tutorial.
Fits my enviroment well, my only question is this:

As I see, the static routes on if-up and if-down restrict the use of the network to the 3 IP addresses, so this type of setup can only be used for ceph communications.

Can I set up VLAN-s over this so that I can give the cluster communications and the VM vlans 10Gbit? Wouldn't live migration kinda break this? I guess I could make a VLAN for cluster comms. But live migrating VMs with their ip moving from machine to machine.... I don't know how that could be possible...

I thought about making simple bridges of the dual 10gbit cards with stp. This should mean that one of the links will stay unused like this A<->B<->C and A<-|->C being unused. I guess I could set up VLANs over this kind of topology, should work for live migrating vm-s too, not?

Does this present any caveats, so that i shouldn't even try this? The traffic between A nd C would go over two hops. Does this extra latency mess with ceph performance?

thx for the insights in advance

CadilLACi

 

[wolfgang]

Hi,

Vlan in an Full Mesh makes no sense, because here is only one network with the same users.
So you win nothing when you separate migration network and cluster network to 2 VLANS.
It makes the setup more complex, lose performance and in case of an failure debugging is very hard.

Also why the vm should use the Full Mesh this can only be use for internal communication.

The easiest way is use a bond like it is described in the wiki.
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_1

 

[CadilLACi]

Wow, thanks for the fast answer!!

So I should do a broadcast mode bond on the dual port 10Gbits, and use that for cluster communication and ceph communication.
The VM-s should use the built in dual 1Gbit interfaces, as usual.

Is that correct?

CadilLACi

 

[wolfgang]

If you have no dedicated cluster network,
I would use for ring0 the full mesh and for ring1 the vm public Network in a VLAN.
https://pve.proxmox.com/wiki/Separate_Cluster_Network

 

[CadilLACi]

Hi!

I successfully set up the mesh network with the method 1 in the full mesh wiki page.

Wasnt able to set up vlans over it, but that doesn't matter now.

I could follow the separate cluster network guidelines too, the cluster communication is running on the mesh network too.

I set up the ceph storage over this network too, doesn't conflict with cluster communications as far as i can tell.

Thx

Laszlo

PS.: does anyone wnat to see pasted config file parts?