Proxmox SDN, openvswitch, and linux bridges

[l0rdraiden]

I want to start a new deploy of proxmox, single server, it will have a "medium" network complexity since it will have a virtual firewall and several vm/LXC

What would the best approach for the network from a security and performance point of view?

What is the status of Proxmox SDN? is already usable and available in the web ui? is faster than linux bridges?

 

[damo2929]

What is the status of Proxmox SDN?
Working stable
is already usable and available in the web ui?
yes when the package is installed on all nodes
is faster than linux bridges?
it's the same as the implementation is using linux bridges

 

[spirit]

about sdn, I'm the main author and I'm using in production (around 4000vms) with evpn. So yes it's stable, and I don't plan to break it

About speed, sdn is using linux bridge, so it's same speed.

It's only a different way to setup them (globally at datacenter level), with also new kind of network support (qinq, vxlan, evpn,...).

New features will coming soon (ipam/dhcp)

 

[VizerDown]

about sdn, I'm the main author and I'm using in production (around 4000vms) with evpn. So yes it's stable, and I don't plan to break it

About speed, sdn is using linux bridge, so it's same speed.

It's only a different way to setup them (globally at datacenter level), with also new kind of network support (qinq, vxlan, evpn,...).

New features will coming soon (ipam/dhcp)

Spirit, In your environment are you just using standard linux bridge/bond networking with SDN and skipping OVS? Trying to learn best practices. Is OVS kind of obsolete with SDN? For us non network monkeys.

 

[spirit]

Spirit, In your environment are you just using standard linux bridge/bond networking with SDN and skipping OVS? Trying to learn best practices. Is OVS kind of obsolete with SDN? For us non network monkeys.

Well, the sdn in proxmox is currently using linux bridge everywhere in background, because you can almost do everything with linux bridge.
(linux bridge= full kernel , ovs = mix kernel + mix ovs service . If the service crash, you loose network).
So, keep it simple.
and for advanced sdn we have bgp-evpn, standard vxlan, standard bgp, and works only with linux bridg.




OVS is used by some sdn controller like OVN , or other openflow controllers. (I known a user working on ovn integration).
Technically, it's possible to create plugin for ovs.

 

[VizerDown]

Well, the sdn in proxmox is currently using linux bridge everywhere in background, because you can almost do everything with linux bridge.
(linux bridge= full kernel , ovs = mix kernel + mix ovs service . If the service crash, you loose network).
So, keep it simple.
and for advanced sdn we have bgp-evpn, standard vxlan, standard bgp, and works only with linux bridg.




OVS is used by some sdn controller like OVN , or other openflow controllers. (I known a user working on ovn integration).
Technically, it's possible to create plugin for ovs.

Thank you

 

[hvisage]

Well, the sdn in proxmox is currently using linux bridge everywhere in background, because you can almost do everything with linux bridge.
(linux bridge= full kernel , ovs = mix kernel + mix ovs service . If the service crash, you loose network).
So, keep it simple.
and for advanced sdn we have bgp-evpn, standard vxlan, standard bgp, and works only with linux bridg.

Is there technical reasons it can't work with OVS, or just a matter of ProxMox made a decision it'll be more cost effective to not also bother with OpenVSwitch for SDN?

OVS is used by some sdn controller like OVN , or other openflow controllers. (I known a user working on ovn integration).
Technically, it's possible to create plugin for ovs.

Hmm.. would like to have a look at that, as I have the need/use of a config server ala AWS, with some IP that is flow-switched. I think OpenStack uses OpenVSwitch for similar.

 

[spirit]

Is there technical reasons it can't work with OVS, or just a matter of ProxMox made a decision it'll be more cost effective to not also bother with OpenVSwitch for SDN?

Well, technically it's possible to implemented ovs plugin. (I'm currently helping a user to create an ovn plugin).

But for my point of view, openflow sdn controllers are dead since end of 201x. Because they are centralized controllers and generaly are not "standard" and can't be integrated with real switch hardware. A lot of openflow vendors are dead too.

That's why I have implemented bgp-evpn, it's standard && decentralized. (as proxmox don't have central node, it's perfect fit).
Also pve-firewall is using iptables/nftables. (That should mean implement a ovs openflow firewall too)


So, The architecture of sdn is open to add your own sdn plugin (or if some vendors want to implement it themself).

 

[hvisage]

Thank you, for explaining the reasoning!

But for my point of view, openflow sdn controllers are dead since end of 201x. Because they are centralized controllers and generaly are not "standard" and can't be integrated with real switch hardware. A lot of openflow vendors are dead too.

Hmmm... interesting statement, given that there are still updates on OpenVSwitch/OVN in the last couple of months with 3.3.0s release.

Which also asks for something else, to understand direction and this comparison of DPDK vs VPP in OVN, is there considerations for supporting VPP/fd.io in ProxMox?

Reason I'm asking for future proofing my current deployments, but also to find how/where I can get some sFlow type output from the ProxMoxs

 

[okela]

about sdn, I'm the main author and I'm using in production (around 4000vms) with evpn. So yes it's stable, and I don't plan to break it

About speed, sdn is using linux bridge, so it's same speed.

It's only a different way to setup them (globally at datacenter level), with also new kind of network support (qinq, vxlan, evpn,...).

New features will coming soon (ipam/dhcp)

Hi spirit, I am looking for a way to implement Proxmox VE in datacenter. If I use SDN on Proxmox, do I also have to config the VXLAN, EVPN,.. on the physical switches (the spine-leaf architecture) ? Does my approach seem OK? I have 10 Proxmox nodes (5 on each rack), I am planning to separate the VMs in each node to different VLANs so I need VXLAN for VMs to communicate to same VLAN but on other rack. Do you have any recommends. Thank you very much!!!

 

[spirit]

Hi spirit, I am looking for a way to implement Proxmox VE in datacenter. If I use SDN on Proxmox, do I also have to config the VXLAN, EVPN,.. on the physical switches (the spine-leaf architecture) ? Does my approach seem OK? I have 10 Proxmox nodes (5 on each rack), I am planning to separate the VMs in each node to different VLANs so I need VXLAN for VMs to communicate to same VLAN but on other rack. Do you have any recommends. Thank you very much!!!

we are using a pure spine/leaf architecture in our datacenter. (only layer3, with point to point switch && bgp between switch && proxmox hosts with dual nic balanced with ecmp).

I don't have support for evpn on my spine && leaf, I'm only doing evpn between our proxmox nodes && our main routers (they are the exit-nodes)

 

[okela]

we are using a pure spine/leaf architecture in our datacenter. (only layer3, with point to point switch && bgp between switch && proxmox hosts with dual nic balanced with ecmp).

I don't have support for evpn on my spine && leaf, I'm only doing evpn between our proxmox nodes && our main routers (they are the exit-nodes)

I dont have experience in Proxmox sdn. Can I use sdn to create vlans in the node and integrate the proxmox nodes to the spine leaf. Or I just simply create linux vlan to use in spine leaf architecture? Does proxmox bridge tag the vlan like a hardware switch? Thank you very much sir!!

 

[spirit]

I dont have experience in Proxmox sdn. Can I use sdn to create vlans in the node and integrate the proxmox nodes to the spine leaf. Or I just simply create linux vlan to use in spine leaf architecture? Does proxmox bridge tag the vlan like a hardware switch? Thank you very much sir!!

a true spine-leaf is only layer3 (routed), so you can't use vlan. you need to use vxlan overlay if you want to propagate same subnet across the differents leafs.

Now, if by "spine-leaf", you mean a classic architecture with core-access switchs with layer2, you can use vlan

https://www.arubanetworks.com/wp-co...pine-leaf-architecture-full-span_1235x567.png

proxmox sdn support both vlan && vxlan.

For vlan, it's working exactly like a real switch, it's tagging ethernet frame with vlan tag.

 

[okela]

a true spine-leaf is only layer3 (routed), so you can't use vlan. you need to use vxlan overlay if you want to propagate same subnet across the differents leafs.

Now, if by "spine-leaf", you mean a classic architecture with core-access switchs with layer2, you can use vlan

https://www.arubanetworks.com/wp-co...pine-leaf-architecture-full-span_1235x567.png

proxmox sdn support both vlan && vxlan.

For vlan, it's working exactly like a real switch, it's tagging ethernet frame with vlan tag.

I saw that you are using Proxmox in production with spine leaf, so you create Vlans and evpn on sdn of Proxmox combine with bgp to get things work? Do you use vxlan and connect your servers to the switchs using Mlag sir?

 

[damo2929]

@spirit I would be interested in a write-up of how your E-VPN stuff works, and would also be interested to know what's needed on real switch

 

[spirit]

I saw that you are using Proxmox in production with spine leaf, so you create Vlans and evpn on sdn of Proxmox combine with bgp to get things work? Do you use vxlan and connect your servers to the switchs using Mlag sir?

if you use layer3 spine/leaf architecture, you need to use vxlan or evpn. (evpn is vxlan + bgp) to create a virtual layer2 network on top, as you want to be able to move/live migrate vm between hosts, and share same subnet/ips between hosts.

@spirit I would be interested in a write-up of how your E-VPN stuff works, and would also be interested to know what's needed on real switch

nothing is needed on real switch. you just need exit-node to route traffic from evpn to real network.

and exit-node can be a proxmox node itself , or if you have a router supporting evpn (cisco, arista, ....) you can use it also.

 

[okela]

if you use layer3 spine/leaf architecture, you need to use vxlan or evpn. (evpn is vxlan + bgp) to create a virtual layer2 network on top, as you want to be able to move/live migrate vm between hosts, and share same subnet/ips between hosts.


nothing is needed on real switch. you just need exit-node to route traffic from evpn to real network.

and exit-node can be a proxmox node itself , or if you have a router supporting evpn (cisco, arista, ....) you can use it also.

If I have 2 spine switches and 2 leaf switches (all support the technology of spine-leaf). So if I configure the switch with spine-leaf techonology (EVPN-vxlan) and just config vlan-aware bridge on Proxmox, so my VMs in cluster 1 can communicate with VMs in cluster 2, right? (no need to use the SDN feature of Proxmox, right?). Thank you sir! (I attach an image for description)