VXLAN Multitenant
- Thread starter Frederico Siena
- Start date
Prior to migrating to proxmox from opennebula, we had assigned a vlan tag for private networking for every customer. Currently, we don't have an excessive amount of customers and private vlans are still an easy way for us to manage inter VM communication.
Thats funny we had been testing open nebula as well and decided it wasn't for us.
Do you mind if i ask what was your reasoning for not staying with OpenNebula? If you would like to keep it off public forum feel free to PM me directly and we can discuss offline.
Was just thinking the same thing about using a vLan for simplicity.
with over 4000 vLans to use it would be a stretch to fill this on a single cluster.
May i ask how are you tracking the vLans?
Do you use any auto provisioning to add a client to a vLan or just manually on request?
""Cheers
G
Do you mind if i ask what was your reasoning for not staying with OpenNebula? If you would like to keep it off public forum feel free to PM me directly and we can discuss offline.
Was just thinking the same thing about using a vLan for simplicity.
with over 4000 vLans to use it would be a stretch to fill this on a single cluster.
May i ask how are you tracking the vLans?
Do you use any auto provisioning to add a client to a vLan or just manually on request?
""Cheers
G
Hi,
I'm working a new sdn module for proxmox 6.0 , it'll be possible to configure virtual network (aka datacenter level configured bridge) with vlan or vxlan.
I'm also looking to implement vxlan with evpn-bgp for 6.1 (each proxmox host have an anycast ip (same ip on each proxmox host bridge), the gateway of the vms), with full layer3 routing stack.
I have write some doc to manually configure vxlan in /etc/network/interfaces
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
but my new sdn module will auto-configure this for you.
I'm working a new sdn module for proxmox 6.0 , it'll be possible to configure virtual network (aka datacenter level configured bridge) with vlan or vxlan.
I'm also looking to implement vxlan with evpn-bgp for 6.1 (each proxmox host have an anycast ip (same ip on each proxmox host bridge), the gateway of the vms), with full layer3 routing stack.
I have write some doc to manually configure vxlan in /etc/network/interfaces
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
but my new sdn module will auto-configure this for you.
That’s very cool!
At the moment we are still in testing phase with PVE but something keeps always coming back to “Keep IT Simple”.
It may be that oVS could be enough to get the job done.
With over 4000 vLans available that’s a lot of clients on a single cluster that can have a private network.
Thanks for the link
I’ll have a look over the documentation when I get a min.What are your expectations with the module availability will it be ready for production in v6 or still in testing.
Would this be exposed via API so that it could be automated?
Very interested
“”Cheers
G
That’s very cool!
At the moment we are still in testing phase with PVE but something keeps always coming back to “Keep IT Simple”.
It may be that oVS could be enough to get the job done.
With over 4000 vLans available that’s a lot of clients on a single cluster that can have a private network.
Thanks for the link
What are your expectations with the module availability will it be ready for production in v6 or still in testing.
Would this be exposed via API so that it could be automated?
Very interested
“”Cheers
G
Currently I'll don't implement ovs, only linux bridge (vlan, vlan-aware, vxlan), with full network reload too.
I can tell you if it'll be production ready for 6.0 , but I'm working on it since 1year now, so it's pretty much stable.
for 6.1, I'm sure it'll be ready.
(I still need to do permissions code, to allow a proxmox user to use only a specific vlan for example)
And of course, all the setup can be done through api. (Proxmox gui use api, so all things you can do in gui, can be done through rest api too)
Interesting I thought oVS was a requirement for vxlan, is this not the case?
We are a VMware house and vDS is the same as oVS with vxlan overlay.
I’d be interested to do some testing with the setup when it is released for sure
So are you part of the ProxMox team or is this a side project?
“”Cheers
G
We are a VMware house and vDS is the same as oVS with vxlan overlay.
I’d be interested to do some testing with the setup when it is released for sure
So are you part of the ProxMox team or is this a side project?
“”Cheers
G
some years ago yes, but now linux kernel support 100% vxlan. (thanks to cumulus linux for the implementation).
The only part missing in debian was the config management in /etc/network/interfaces,
but thanks to cumulus linux again, we have now ifupdown2 package which manage a lot of new network features
(I'm not a big fan of ovs
I'm a contributor on proxmox, and need this feature for my company. (we are going to full vxlan next year on top of a layer3 network).
I don't like to maintain patch in a side project, so I'm pushing to implement this officialy in proxmox.
(So yes, it'll be official, and some code is already commited for proxmox6)
Hi Spirit.. Do you have the code available for testing? We've been looking to VXLAN paired up with PVE for the past few months. Great to know you're working on it and there's a path for native integration with the next release.
some years ago yes, but now linux kernel support 100% vxlan. (thanks to cumulus linux for the implementation).
The only part missing in debian was the config management in /etc/network/interfaces,
but thanks to cumulus linux again, we have now ifupdown2 package which manage a lot of new network features
(I'm not a big fan of ovs
I'm a contributor on proxmox, and need this feature for my company. (we are going to full vxlan next year on top of a layer3 network).
I don't like to maintain patch in a side project, so I'm pushing to implement this officialy in proxmox.
(So yes, it'll be official, and some code is already commited for proxmox6)
Hi,
it's not yet ready, and I'll don't have time to finish it for proxmox 6.0. (I'm targetting proxmox 6.1)
I have already write some doc to do it manually in /etc/network/interfaces with ifupdown2
https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD
Hi Spirit
if doing this manually will Proxmox pick up the config once its in 6.1?
I'm patient and can wait if its slotted for end of year or even early next year, would prefer to get it right from the get go for a smoother roll out.
""Cheers
G
no sorry.
the config will be generated in a separate /etc/network/interfaces.d/X.
(the main idea will be to defined networks at datacenter level in a proxmox format (/etc/pve/xxx.cfg), then the local config will be generated on each node /etc/network/interfaces.d/x and reloaded online).
Hi Spirit
thanks for replying
thats cool and as mentioned i'm super happy to wait, have lots of experience with VMware VXLAN and am very aware of the intricacies of the technology and how it functions, no rush to manually start down a path which could end up in down time at the end of the day for clients when production environment comes around.
happy to help with anything i can be of assistance with (my knowledge is limited but happy to help where i can) - sing out if you need anything.
looking forward to the final release.
have an awesome day.
""Cheers
G
thanks for replying
thats cool and as mentioned i'm super happy to wait, have lots of experience with VMware VXLAN and am very aware of the intricacies of the technology and how it functions, no rush to manually start down a path which could end up in down time at the end of the day for clients when production environment comes around.
happy to help with anything i can be of assistance with (my knowledge is limited but happy to help where i can) - sing out if you need anything.
looking forward to the final release.
have an awesome day.
""Cheers
G
Hi Spirit
thanks for replying
thats cool and as mentioned i'm super happy to wait, have lots of experience with VMware VXLAN and am very aware of the intricacies of the technology and how it functions, no rush to manually start down a path which could end up in down time at the end of the day for clients when production environment comes around.
happy to help with anything i can be of assistance with (my knowledge is limited but happy to help where i can) - sing out if you need anything.
looking forward to the final release.
have an awesome day.
""Cheers
G
cool, thanks
BTW, do you use nsx ?
(Do you want only vxlan between vm, or also vxlan routing like nsx ? (because It's planned too, with frr routing+bgp-evpn)
Hay Mate
we do use NSX in our VMware deployment.
NSX Edges and NSX master.
What NSX features are in the planning?
We are looking at an alternative to NSX Edge devices still trying to work out the best option for users at this stage no solid ideas - have been looking at Opensense and Pfsense etc as a starting point.
Are Edges on the plan?
""Cheers
G
we do use NSX in our VMware deployment.
NSX Edges and NSX master.
What NSX features are in the planning?
We are looking at an alternative to NSX Edge devices still trying to work out the best option for users at this stage no solid ideas - have been looking at Opensense and Pfsense etc as a starting point.
Are Edges on the plan?
""Cheers
G
It's a little bit different than nsx, because you don't need edges
Each vxlan bridge on proxmox nodes have the same anycast ip (and it's the gateway for the vm), then you can route to your external router. (with static route or bgp ). (So each proxmox node is the edge)
Another way, is that if your router support bgp-evpn, they can directly announce the default route, and proxmox nodes will auto forward packets through vxlan to the router.
I known that nsx support also proxy (through an haproxy vm), it's not planned.
For future release, I'll like to implement dhcp and network features.
If you have a specific nsx feature in mind, don't hesitated to share it , I'll look if it's possible to implement it.
Each vxlan bridge on proxmox nodes have the same anycast ip (and it's the gateway for the vm), then you can route to your external router. (with static route or bgp ). (So each proxmox node is the edge)
Another way, is that if your router support bgp-evpn, they can directly announce the default route, and proxmox nodes will auto forward packets through vxlan to the router.
I known that nsx support also proxy (through an haproxy vm), it's not planned.
For future release, I'll like to implement dhcp and network features.
If you have a specific nsx feature in mind, don't hesitated to share it , I'll look if it's possible to implement it.
To be straight up i personally like simple.
The simpler a feature is, the better it works!
There have been way too many issues over the years with over complicated features from VMware what are hard to track which mens hard to fix.
VXlan by design ticks all the boxes we are looking for and with integrated ProxMox management it's a perfect fit for our roadmap.
i'll have a think about NSX a little more and get back to you if anything pops up that is a good value proposition.
thanks for thinking big picture and keeping us in the loop it's greatly appreciated
""Cheers, speak soon.
G
The simpler a feature is, the better it works!
There have been way too many issues over the years with over complicated features from VMware what are hard to track which mens hard to fix.
VXlan by design ticks all the boxes we are looking for and with integrated ProxMox management it's a perfect fit for our roadmap.
i'll have a think about NSX a little more and get back to you if anything pops up that is a good value proposition.
thanks for thinking big picture and keeping us in the loop it's greatly appreciated
""Cheers, speak soon.
G
Is there any way to make this work with the "unicast mode" and vlan_aware shown here: https://git.proxmox.com/?p=pve-docs.git;a=blob_plain;f=vxlan-and-evpn.adoc;hb=HEAD?
Currently this fails on my side
Currently this fails on my side