Proxmox Cluster Networking question?

Discussion in 'Proxmox VE: Networking and Firewall' started by killmasta93, May 3, 2019.

  1. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    436
    Likes Received:
    16
    Hi,

    i was wondering if someone could shed some light on what im trying to do. Currently have Site to site Openvpn working great no issue. what im trying to do on Site B i have a cluster on proxmox, the idea is that i want to take one of the servers to Site A, but the problem is that it has a different Ip from Site A, would it be possible to move that server to Site A and somehow keep the same IP and be able to communicate with the Site B with the site to site?

    im trying to wrap my head around it not sure if i need to VLAN on the edge swich or how could i approach this?


    https://imgur.com/a/ABi0aMy



    Thank you
     
  2. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    315
    Hi,

    Do you try to run a cluster over a VPN?
    If yes this does not work.
    The latency is too high and multicast does not work over VPN tunnels that easy.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    436
    Likes Received:
    16
    Thanks for the reply, currently i have the cluster not though Site to site, the idea is to have only 1 node on the other side of the site to site VPN. as for the latency not sure would a problem, my issue is that node from site A has a different subnet from Site B which i was reading need VXLAN and OVS bridge. I put the OVS bridge working but not sure whats the next step to VXLAN
     
  4. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    315
    If your latency is higher than 7 ms the problems will start in the hole cluster not only on the one outside.
    Do yourself a favor and make a long time benchmark to prove if you always under the 7ms.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    436
    Likes Received:
    16
    Thanks for the reply, well your right the ms im getting is 20-28 max when you mean hole not sure what you mean?
     
  6. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,763
    Likes Received:
    315
    Proxmox VE cluster uses a real-time filesystem to keep information and states.
    If one node is always flapping it will disturb the cluster filesystem and you will not able to manage it.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    436
    Likes Received:
    16
    would it be and issue if i throw the cluster for that node with the WAN ip? instead of the LAN?
     
  8. crainbramp

    crainbramp New Member

    Joined:
    Apr 11, 2019
    Messages:
    5
    Likes Received:
    1
    As a CSP who runs Proxmox clouds for a living, I can tell you ... this is a BAD idea.

    The problem is, that the cluster network is VERY sensitive to latency. This is why it's recommended, and really a necessity for production clusters, that your cluster traffic be separated out on it's own redundant bond by itself. We run 8 NICs per node to get the redundancy and latency that we need.

    Once your latency gets over that threshold (7ms-10ms) the ENTIRE cluster will start having issues. Not just the "remote" node in this case. It can even get so bad that the cluster will start rebooting nodes, and if you're thinking about HA stretched across DCs ... just dont. The only customer we have doing this successfully does it between our Seattle Data Center and Amazon on a direct-connect line with <3ms latency, and buddy, they PAY for that connection.

    As Always,
    Crain
     
    killmasta93 likes this.
  9. wirdo02

    wirdo02 New Member
    Proxmox Subscriber

    Joined:
    Mar 25, 2019
    Messages:
    8
    Likes Received:
    1
    I've been struggling with the same issues you are right now.
    After beeing in contact with the proxmox team I went with pve-zsync instead. It does require a manual action to turn on the VM on the other side though. I made a small webpage for my collegues to enable the VM in case our main location goes down.

    Probably not the prettiest way to do it but at least it's stable this way.
    Every 15 minutes a replication is being made to the offside location.

    Also if I'm correct; looking at your graph you would have 3 servers in one location and 1 in the other.
    If side B fails you have no quorum and will result in your entire cluster going on halt :)
     
    killmasta93 likes this.
  10. killmasta93

    killmasta93 Member

    Joined:
    Aug 13, 2017
    Messages:
    436
    Likes Received:
    16
    yeah correct i ended up using the server on site B off the cluster and just use pve-zsync to backup the VMs
     
    crainbramp likes this.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice