Best configuration for a cluster-wide virtual switch

[kavejo]

Hi,

I have recently set-up a Proxmox VE cluster with 2 HPE DL380e Gen. 8.

I’m now looking at the best way to have a segmented virtual network that could span the cluster nodes, so to ensure VMs on the same LAN can communicate to each other regardless of the host where they reside.

I have set-up vmbr0 so that it’s linked to eno1, this is the management network on both nodes, then I have vmbr1 which is bound to bond0 that is linked to eno2 and eno3 and that is used for cluster and replication and lastly vmbr2 which is linked to eno4.

All these NICs are connected to the same physical switch (HP 1810-48G) that is serving a 192.168.1.0/24 network. I would ideally like to find a way to have all the vmbr2 traffic segmented from the other traffic and ideally also use a different address space (i.e. 10.0.0.0/24); is there any easy way to achieve what I’m after?

I don’t need clients on vmbr2 to have internet connectivity, I just need to have VMs on different hosts to be able to communicate across these hosts. I would deploy a pfSense or vyOS or a Sophos UTM appliance with a WAN NIC on vmbr0 and the LAN NIC ok vmbr2.

Basically I would need to achieve a setup similar to a VMware Distributed vSwitch, where the vNetwork is isolated from the physical switch and downs the hosts.

What would be the best way to do so, supposing this is achievable?

Thank you very much!

 

[102020]

Hey there, so basically you are in the same boat as I am, only difference is I've setup a mesh network instead of using a switch on the backbone for the VXLAN.

You want to be looking at openvswitch for your layer 3 switching, and pfSense to connect your vmbr0 to vmbr1/2/3/etc. This way you can segregate off with vlans, different MTUs, etc.

 

[kavejo]

Thank you @102020 for your reply.

My question is more on "how do I do so?".
I am familiar with the steps required to create a Distributed vSwitch in VMware but I am finding it user unfriendly to achieve in Proxmox VE.

I mean, assuming I reserve eno4 for vmbr2 for this purpose, how can I ensure host on the same virtual network residing on different hosts talk to each other?

I mean, if VM1 (10.0.0.1) runs on HOST1 (192.168.1.1) and VM2 (10.0.0.2) runs on HOST2 (192.168.1.2), assuming both VMs are linked to vmbr2 which exists with the same name and comment on both servers (and which has no IP assigned), how I can make sure they talk with each other but they don't talk with the reminder hosts connected to the same physical switch?

Do I need to configure Open vSwitch? If so, what configuration should I use, proven that I would like to leave the other NICs configured as they are?

I am kind of looking at https://documentation.online.net/en/dedicated-server/tutorials/network/rpn-proxmox-openvswitch which seems to infer on one host there is a need for referencing the IP of the other host, but I am not finding it extremely explanatory.

Thanks!

 

[102020]

Need a little bit more detail before I start blabing out details.
How many hosts do you have, how many NICs, which are WAN facing, which are VXLAN facing?
The GRE tunnel is an option, but it really comes down to how you have this deployed and then I can probably guide you from there.

 

[kavejo]

Hi @102020,

Please let me share here the details.

This setup consists of 2 equally spec’d servers, both DL380e Gen.8 and each have 4 NICS. The network setup is the same on both boxes and all 4 NICs, for each server, are connected to the same physical switch (192.168.1.253) which provides access to router (192.168.1.254) and therefore to internet.

In both servers eno1, the primary NIC is used for management, while 2 NICs configured in LACP are used for the cluster, the reminder is connected but unconfigured.

SRV1:
- eno1 - vmbr0 - 192.168.1.31
- eno2 - bond0 - 192.168.1.32
- eno3 - bond0 - 192.168.1.32
- eno4 - unconfigured

SRV2:
- eno1 - vmbr0 - 192.168.1.41
- eno2 - bond0 - 192.168.1.42
- eno3 - bond0 - 192.168.1.42
- eno4 - unconfigured

What I would like is to have eno4, on both servers, to be used for routing the virtual network traffic across the hosts. Ideally I would want this traffic to be in the 10.0.0.0/8 range and I would assign static IPs in this range to the VMs. This network does not need to have a path to internet, I would deploy one routing VM (vyOS, pfSense or similar) with a NIC on vmbr0 and one one the virtual network.

Thank you!

 

[kavejo]

Good morning all,

Browsing the forum I found this article https://forum.proxmox.com/threads/proxmox-vm-lan-between-nodes.63049/.

I am wondering if the same configuration could be applied to my set-up.
I could set eno4.100 (last interface, VLAN 100) on both nodes and set the switch to only allow traffic tagged for VLAN 100 on these 2 interfaces.
Not sure that is the best way though - I would appreciate some suggestions if anyone has a similar setup.

Thank you.

 

[kavejo]

That didn’t seem to work well for me or, well, I must not have implemented fully.

What I want to achieve is basically this: https://ilearnedhowto.wordpress.com...n-vswitch-in-order-to-connect-lxc-containers/.

Any suggestion on the best (modern) way to do so?

 

[kavejo]

For posterity, I seem to have resolved the problem by adding the following 2 entities on /etc/netweok/interfaces:

auto vxlan0
iface vxlan0 inet manual
        vxlan-id 2
        vxlan_remoteip 192.168.1.**

auto vmbr2
iface vmbr2 inet manual
        bridge_ports vxlan0
        bridge_stp off
        bridge_fd 0
#Distributed Virtual Network

The IP on the vxlan_remoteip attribute is the one from bond0 to which I have added the last interface (eno4) as well.

All that is left to do now is to make sure the traffic flows from bond0 to bond0 and not from eno1 to bond0 and possibly set the MTU to something like 1450 for the VXLAN traffic.

 

[spirit]

for mtu, indeed, you have 2 possibility:
increase mtu of physical interface (1550).
If you can't, you need to decrease it to 1450 on bridge, but also in your vm guest os network config.

also if you need multiple vxlan, it possible to do something like

%for v in range(1010,1021):
auto vxlan${v}
iface vxlan${v}
        vxlan-id ${v}
        bridge-access ${v}
        vxlan_remoteip 192.168.1.*
        vxlan_remoteip 192.168.1.*  
%endfor


auto vmbr2
iface vmbr2 inet manual
        bridge_ports glob vxlan1010-1020
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

this is with a vlan aware bridge, so vlan setup in vm nic configuration, will be mapped to vxlanid.

the "for" loop is optionnal, this is to avoid to write each vxlan in your /etc/network/interfaces


I'm currently working on proxmox sdn feature, vxlan supported will be present for this kind of config.

 

[kavejo]

Hi @spirit,

Thank you for the information - it seems very cool that Software Defined Network could be included in Proxmox.
Do you know when such feature would be available?

Would you require any help with test this feature? If so I would be more than happy to run some tests.

Thank you.

 

[spirit]

Hi @spirit,

Thank you for the information - it seems very cool that Software Defined Network could be included in Proxmox.
Do you know when such feature would be available?

I'm targeting proxmox 6.2.

Would you require any help with test this feature? If so I would be more than happy to run some tests.

Yes ! help is welcome. can you send me an email directly ? (aderumier@odiso.com)

 

[kavejo]

Just emailed you on the back of the automated Proxmox forum email that notified me of your response