3 node mesh network VLAn-able?

Discussion in 'Proxmox VE: Networking and Firewall' started by CadilLACi, Feb 3, 2017.

  1. CadilLACi

    CadilLACi New Member

    Joined:
    Sep 20, 2016
    Messages:
    14
    Likes Received:
    0
    Hello,

    I have 3 servers with dual 10Gbit cards, but no 10g switch.

    I read the https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server tutorial.
    Fits my enviroment well, my only question is this:

    As I see, the static routes on if-up and if-down restrict the use of the network to the 3 IP addresses, so this type of setup can only be used for ceph communications.

    Can I set up VLAN-s over this so that I can give the cluster communications and the VM vlans 10Gbit? Wouldn't live migration kinda break this? I guess I could make a VLAN for cluster comms. But live migrating VMs with their ip moving from machine to machine.... I don't know how that could be possible...

    I thought about making simple bridges of the dual 10gbit cards with stp. This should mean that one of the links will stay unused like this A<->B<->C and A<-|->C being unused. I guess I could set up VLANs over this kind of topology, should work for live migrating vm-s too, not?

    Does this present any caveats, so that i shouldn't even try this? The traffic between A nd C would go over two hops. Does this extra latency mess with ceph performance?

    thx for the insights in advance

    CadilLACi
     
  2. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    316
    Hi,

    Vlan in an Full Mesh makes no sense, because here is only one network with the same users.
    So you win nothing when you separate migration network and cluster network to 2 VLANS.
    It makes the setup more complex, lose performance and in case of an failure debugging is very hard.

    Also why the vm should use the Full Mesh this can only be use for internal communication.

    The easiest way is use a bond like it is described in the wiki.
    https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Method_1
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. CadilLACi

    CadilLACi New Member

    Joined:
    Sep 20, 2016
    Messages:
    14
    Likes Received:
    0
    Wow, thanks for the fast answer!!

    So I should do a broadcast mode bond on the dual port 10Gbits, and use that for cluster communication and ceph communication.
    The VM-s should use the built in dual 1Gbit interfaces, as usual.

    Is that correct?

    CadilLACi
     
  4. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    316
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. CadilLACi

    CadilLACi New Member

    Joined:
    Sep 20, 2016
    Messages:
    14
    Likes Received:
    0
    Hi!

    I successfully set up the mesh network with the method 1 in the full mesh wiki page.

    Wasnt able to set up vlans over it, but that doesn't matter now.

    I could follow the separate cluster network guidelines too, the cluster communication is running on the mesh network too.

    I set up the ceph storage over this network too, doesn't conflict with cluster communications as far as i can tell.

    Thx

    Laszlo

    PS.: does anyone wnat to see pasted config file parts?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice