Proxmox Ceph Redundant Network Setup help required

Discussion in 'Proxmox VE: Networking and Firewall' started by Mario Minati, Jun 12, 2018 at 00:45.

  1. Mario Minati

    Mario Minati New Member

    Joined:
    Jun 11, 2018
    Messages:
    2
    Likes Received:
    0
    Hello @all,

    we are new to Proxmox. Currently we are using Univention Corporate Server to virtualize 15 machines with 3 physical servers. We are lacking a shared storage and HA. Therefore we would like to setup a proxmox cluster with 5 physical machines, 3 identically configured machines for ceph and 2 machines for virtualisation.

    We read a lot of posts in the forum, the wiki, the docs and thing we have enough background to start setting things up. We would like to ask if out network setup is suitable for the goals of high availabilty and redundance.

    Attached you find a diagram of our network setup:
    - 2 seperate 1 GBE networks for coro sync ring 0 and ring 1 with seperate switches from which we use 1 network for management (external access to proxmox web interfaces and lights out management)
    - 2 seperate 10 GBE networks as ceph public networks with seperate switches and usage bonding
    - 2 seperate 10 GBE network as ceph cluster network with seperate switches and usage of bonding
    - 1 seperate 1 GBE network to access the virtual machines from the outside (DMZ / Intranet)

    Questions:
    - Is this suitable for redundancy?
    - Is this suitable for good performance?
    - Is the selected bond_mode (balance_rr) ok for use in a configration with seperate switches to acchieve also a good performance?

    Thanks for your suggestions!

    Best greets,

    Mario Minati
     

    Attached Files:

  2. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,278
    Likes Received:
    110
    +1 or :thumbsup: ; Depending on the needs of your backup (backup/iso/templates), you may need more than 1 GbE.

    Looks good on the redundancy level, but check the latency, Ceph is very sensitive to latency and as lower the better.

    If 1 GbE is sufficient, enough for peak traffic? This isn't redundant, is it?

    First, +1 for the nice network diagram. For redundancy, my comments above. The balance_rr mode will send TCP packets out of order as traffic increases, this will trigger a retransmit and stall your ceph network. Better use a active+backup or LACP. For connecting two switches you may need MLAG. I guess a "easier" setup might be to use a 2x10 GbE bond on both switches and separat ceph's public and cluster network through VLAN and a active+backup setup that utilizes each bond individually, up to a switch failure. On failure, both networks would be put on one bond. This still keeps redundancy, but you don't need MLAG or any other method to have inter-switch LACP.
     
  3. Mario Minati

    Mario Minati New Member

    Joined:
    Jun 11, 2018
    Messages:
    2
    Likes Received:
    0
    Hello Alwin,

    thanks for your advices. We improved our network setup according to your suggestions (new network diagram is attached) :

    - The management net ist now seperated from corosync ring 0 net, we use additional 10GBE network ports, so we can change the exisiting 1 GBE network switch if we get bandwidth problems on that network

    - To add redundancy and improve peak bandwidth to the outer world (1 GBE dmz network) we use an additional 10 GBE network port in bonding configuration with an 1 GBE port. Is the bond_mode balance_rr suitable for that connection or should it also be an active-backup bond?

    - The bond mode for ceph private and public networks are changed to active-backup, but we still would like to use seperated switches, which should provide use with the desired redundancy, right? I personally dont't like using VLAN that much, as it offers one more step of complexity where we can make mistakes.

    After setup of ceph private and public network we will check latency with the test commands given in the docs... We expect low latency.

    If you would like our kind of network diagram for documentation we can provide you with the LibreOffice Draw file. :)


    Best greets,

    Mario Minati
     

    Attached Files:

  4. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    1,278
    Likes Received:
    110
    Good, corosync separation will save the bacon. ;)

    As written on the post above, as the packets might be out-of-order, the network card has the extra job to put all packets in sequence again. This may or may not work, depending on your application. If you have a 10 GbE connection then a active-backup with primary 10 GbE would not only give you more bandwidth but also lower latency.

    I guess, my description was a little bit confusing. In the case you can afford extra interfaces on both machines, then you don't need my idea. But if you want redundancy with no extra hardware, then the idea is as follows.

    | eth0.100 & eth1.100 => bond0 => primary (eth0) on switch1 (cluster)
    | eth0.101 & eth1.101 => bond1 => primary (eth1) on switch2 (public)

    I hope it illustrates what I meant. On failure both, public & cluster reside on the same member of the bond. At normal operation both run separated.

    You may compare your results with our benchmarks and comparisons from other users in the thread.
    https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/

    Thanks for the offer, but I must decline. Definitely a good reference for the discussion though.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice