proxmox 7.0 sdn beta test

[spirit]

Spoiler: frr.conf

log syslog informational
ip forwarding
ipv6 forwarding
frr defaults datacenter
service integrated-vtysh-config
hostname pve02
!
!
vrf vrf_public
 vni 10000
exit-vrf
!
router bgp 65000
 bgp router-id 192.168.10.2
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp network import-check
 no bgp ebgp-requires-policy
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor 192.168.10.250 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 192.168.128.1 peer-group VTEP
 neighbor 192.168.128.3 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_public
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65000 vrf vrf_public
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
line vty

can your tri to edit frr.conf, and add network X.X.X.X/X in

 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family

 address-family ipv4 unicast
  network X.X.X.X/X
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family

where X.X.X.X/X is the subnet of your evpn. (ex: "network 192.168.0.0/24")
then restart frr service.


because, I'm not sure than the routes coming from evpn (with the "import vrf_public"), are automaticaly announced.

 

[spirit]

another thing to try, instead "network X.X.X.X/X", try to add "redistribute connected"

 address-family ipv4 unicast
  redistribute connected
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family

 

[forsytheda]

Neither of those seemed to work in getting my vm to connected to anything outside of the zone. I am seeing the networks from my router in the route table on pve02, and I see the networks for pve02 in the router, but they are unable to connect. I also am unable to ping the vnet gateway from the proxmox host itself, so that may be there the problem lies.

 

[spirit]

I am seeing the networks from my router in the route table on pve02, and

do you see networks that you want to reach from your vm ?

I see the networks for pve02 in the router,

do you mean the evpn network subnets ? if yes, that's ok the the bgp part ? (is it without "redistributed connected" or "network .X.X.X.X" ?)

can you send the result on pve2 of

#vtysh
   -->show ip route

but they are unable to connect.

can you try to ping from your vm to outside,

the do a "tcpdump -i vrf_public" to see traffic inside the vrf.

can you try to do it on pve2, to see if traffic is coming from the vm node.

tcpdump "tcpdump -i ethX" on pve2 (where ethx is the interface where the traffic to opense is going through)

I also am unable to ping the vnet gateway from the proxmox host itself, so that may be there the problem lies.

this is expected, the evpn network is in a different vrf. only the exit node have a leak between the vrf to able to route between them.
and the vnet gateway is really special, as it's an anycast gateway. (same ip can be on differents hosts). So it's only respond to the vm directly connected.

 

[Nascire]

I think I found two "bugs":
SDN itself - Unable to remove Subnet if Gateway specified (makes kind of sense, but the error message is therefore misleading)

PHPIPAM - Unable to specify Gateway - digging makes it seem that the "1" for is_gateway cannot be translated/is not transmitted

 

[spirit]

I think I found two "bugs":

thanks for the repot

SDN itself - Unable to remove Subnet if Gateway specified (makes kind of sense, but the error message is therefore misleading)

I'll try to autoremove gateway from ipam, on subnet removal, if gateway is the last ip from subnet.

PHPIPAM - Unable to specify Gateway - digging makes it seem that the "1" for is_gateway cannot be translated/is not transmitted

can you send me content /etc/pve/sdn/zones.cfg , /etc/pve/sdn/vnets.cfg, /etc/pve/sdn/subnets.cfg , /etc/pve/sdn/ipams.cfg ?

 

[Nascire]

can you send me content /etc/pve/sdn/zones.cfg , /etc/pve/sdn/vnets.cfg, /etc/pve/sdn/subnets.cfg , /etc/pve/sdn/ipams.cfg ?

Yeah, no problem, but I don´t think that I´ve done anything wrong.
I edited the /usr/share/perl5/PVE/Network/SDN/Ipams/PhpIpamPlugin.pm and added the parameters specified to the error message (line 148 - die "ipam .....")
All parameters showed up as expected, but is_gateway was empty - handing the same call with Postman, the subnet creation and also the addition of the gateway where successful.

On line 147 in /usr/share/perl5/PVE/Network/SDN/SubnetPlugin.pm, from my PoV, the call seems correct, but maybe I´m wrong.

root@pve01:~# cat /etc/pve/sdn/*.cfg
phpipam: ipam
        section 3
        token redacted
        url https://ipam.home.xxxxx.site/api/proxmox

pve: pve

subnet: Trunk-192.168.2.0-24
        vnet V0002

subnet: Trunk-192.168.3.0-24
        vnet V0003

subnet: Trunk-172.20.1.0-24
        vnet V0090

vnet: V0002
        zone Trunk
        alias redacted
        tag 2

vnet: V0003
        zone Trunk
        alias redacted
        tag 3

vnet: V0090
        zone Trunk
        alias redacted
        tag 90

vlan: Trunk
        bridge vmbr0
        ipam ipam

 

[spirit]

All parameters showed up as expected, but is_gateway was empty - handing the same call with Postman, the subnet creation and also the addition of the gateway where successful.

I think you are right

n line 147 in /usr/share/perl5/PVE/Network/SDN/SubnetPlugin.pm, from my PoV, the call seems correct, but maybe I´m wrong.

yes, but I totally forget to add $is_gateway in add_ip in Subnets.pm . (I think it was lost during some merge)


Don't have tested it yet, but this should works

diff --git a/PVE/Network/SDN/Subnets.pm b/PVE/Network/SDN/Subnets.pm
index 46d9830..0231822 100644
--- a/PVE/Network/SDN/Subnets.pm
+++ b/PVE/Network/SDN/Subnets.pm
@@ -232,7 +232,7 @@ sub next_free_ip {
 }
 
 sub add_ip {
-    my ($zone, $subnetid, $subnet, $ip, $hostname, $mac, $description) = @_;
+    my ($zone, $subnetid, $subnet, $ip, $hostname, $mac, $description, $is_gateway) = @_;
 
     return if !$subnet || !$ip; 
 
@@ -259,7 +259,7 @@ sub add_ip {
        my $plugin = PVE::Network::SDN::Ipams::Plugin->lookup($plugin_config->{type});
 
        eval {
-           $plugin->add_ip($plugin_config, $subnetid, $subnet, $ip, $hostname, $mac, $description);
+           $plugin->add_ip($plugin_config, $subnetid, $subnet, $ip, $hostname, $mac, $description, $is_gateway);
        };
        die $@ if $@;
     }

I don't have time to check this until friday, I'll keep you in touch

 

[Nascire]

Don't have tested it yet, but this should works


I don't have time to check this until friday, I'll keep you in touch

That did the trick

First try was without deleting the IP from IPAM, resulting in error 409 object already exists.
Afterwards I deleted it from IPAM and tried again, finally got it working.

In case of the gateway IP, maybe it would be good to update the entry in IPAM (or at least, ask if you want to)?

 

[spirit]

That did the trick

First try was without deleting the IP from IPAM, resulting in error 409 object already exists.
Afterwards I deleted it from IPAM and tried again, finally got it working.

ok, great! I'll send a patch next week to pve-devel mailing list.

In case of the gateway IP, maybe it would be good to update the entry in IPAM (or at least, ask if you want to)?

Yes, I will check that.

Thanks again for testing !

 

[ph0x]

I setup a virtual test cluster yesterday since I'm very interested in this sdn feature and for now it looks pretty promising.
Can you point me towards some resources where I can read about best practices? For example, I have setup three zones (as in three "data centers") and several vtnets inside the zones (as in traditional VLANs), is this the way to think of it?
Am I right that I would then need to setup gateway VMs to route between those vtnets and zones?
One thing that I noticed: the "Alias" tag of the vtnets, which is defined in the sdn section does not get propagated to the zone sections (zone->content) of the participating hosts, that field is just empty.

 

[spirit]

I setup a virtual test cluster yesterday since I'm very interested in this sdn feature and for now it looks pretty promising.
Can you point me towards some resources where I can read about best practices?

The current ressources are in
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

but I think I should add more examples

For example, I have setup three zones (as in three "data centers") and several vtnets inside the zones (as in traditional VLANs), is this the way to think of it?

mmm, no. A zone is more a security zone, than a physical zone. Vnets from a zone should not be able to communicate with vlans in other zone.
It's also possible to give permission to a user on a specific zone, and He will see only vnets from this zone.
For QinQ for example, you have a main vlan on the zone, and sub vlans on vnets.
for bgp-evpn (routage vxlan), a vxlan from a zone can be routed to a vxlan on other zone.
For vlans, a gateway in a zone should not be able to route vlans from others zones

Am I right that I would then need to setup gateway VMs to route between those vtnets and zones?

yes, the gateway is not yet implemented in proxmox. (but it should come soon, currently in discussion).
so you need to implement your own gateway (vm or physical, what you want).

Basicaly, for vlan, it's the same than define them locally on each server vmbrX, but it's done once at datacenter level

One thing that I noticed: the "Alias" tag of the vtnets, which is defined in the sdn section does not get propagated to the zone sections (zone->content) of the participating hosts, that field is just empty.

The alias is really just an alias/description (no characters limit) of the vnet (limited to 8 characters). I'll be display later in the vm nic gui when you choose the bridge.

 

[spirit]

@Nascire I have send patch to pve-devel mailing list to fix the is_gateway, and I also allow to reuse the gateway ip if it's already in ipam with "is_gateway" flag. Like this, it's possible to redeclare subnets on multiple proxmox clusters sharing the same ipam.

I'll should be available soon in new package version.

Thanks again for reporting this bug.

 

[woelfl]

Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:


Bug?

 

[spirit]

Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:
View attachment 26617

Bug?

Hi, mmm, I think I have forgot something in the last version. (I don't have worked too much in permissions, I still need to add pool permissions too)
I'll check that today or tomorrow, thanks for report this.

 

[ph0x]

mmm, no. A zone is more a security zone, than a physical zone. Vnets from a zone should not be able to communicate with vlans in other zone.

Yeah, understood. That was my intention, already assuming that "datacenter" might bring in the wrong notion.

yes, the gateway is not yet implemented in proxmox. (but it should come soon, currently in discussion).
so you need to implement your own gateway (vm or physical, what you want).

Check.

The alias is really just an alias/description (no characters limit) of the vnet (limited to 8 characters). I'll be display later in the vm nic gui when you choose the bridge.

Check so far, but when I view a zone under a host, there already is an uneditable column for the Alias, which - I assume - should be filled with the Alias definition of the SDN section as well.

 

[spirit]

Check so far, but when I view a zone under a host, there already is an uneditable column for the Alias, which - I assume - should be filled with the Alias definition of the SDN section as well.

oh, you're right. I forgot to add the content. I'll fix that asap, thanks for reporting !

 

[spirit]

Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:
View attachment 26617

Bug?

I have send a fix the pve-devel mailing.
It seem to have breaked recentyly

diff --git a/src/PVE/AccessControl.pm b/src/PVE/AccessControl.pm
index 888875e..4754c5c 100644
--- a/src/PVE/AccessControl.pm
+++ b/src/PVE/AccessControl.pm
@@ -945,6 +945,7 @@ sub check_path {
        |/pool
        |/pool/[[:alnum:]\.\-\_]+
        |/sdn
+        |/sdn/zones/[[:alnum:]\.\-\_]+
        |/storage
        |/storage/[[:alnum:]\.\-\_]+
        |/vms

It's just missing a path check in accesscontrol, you can try to add the missing line manually in /usr/share/perl5/PVE/AccessControl.pm ,
and restart pveproxy daemon.

 

[eduncan911]

I apologize for the newbie questions, as I am trying to wrap my head around Proxmox's SDN vs the rack I am setting up for a school. I would like to list out the basic hardware, and the planned isolation and L3 switches.

• There will be 5x hosts running Proxmox, all fairly equal hardware.
• Ceph will be the backend, across all 5 nodes (OSDs on each node).
• Each host has dual 10G fiber, and dual 1G copper.
• 2x Brocade Enterprise L3 switches
  • Stacked and setup for fail-over/mirror/LAG across the switches.
  • Each host will have a single 10G and a single 1G to one switch, and a LAG setup to the other switch with another 10G and 1G
  • The concept is, if I reboot a switch, they are still up and running on the other switch.
  • 4x VLANs each: Public + Ceph-data over 10G LAG; Corosync + Management over 1G LAG

The requirements for the school are:

• A few backend VMs for the teachers, mostly shared VLAN networking is fine.
• Isolated VLAN management of hosts, switches, etc.
• One "public-facing" app on a static IP (that I am not happy/fully trust).

It's that last one that landed me on this SDN idea for Proxmox. Normally, I would setup some sort of DMZ (between two NATs) to isolate the box from the backend private network.

My plan was to carefully create yet another VLAN dedicated to a virtual DMZ I'll create within the switches - with a NAT gateway running within another VM - back to the switch. This just feels dirty though. Especially if they were able to break out of the VM and onto the Proxmox host... I do not have another "server" to run as a separate gateway/NAT box for that DMZ. Hence, why I am planning on that odd combination.

Would Proxmox SDN help provide a solution here?

Especially since I have Enteprise L3 switches at the ready that supports:

• Full layer 3: IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, etc
• BGP, VRFs, tunnels, everything - all high-end L3 switching protocols
• OpenFlow in hardware for SDN, including hybrid port mode
• MACSEC on the SFP+ ports for 80gbps of real time L2 AES-128 encryption

In the end, I prefer a simplified interface vs the in-switch ASIC hardware configurations. And it sounds like this Proxmox SDN implementation with OVS to manage the VLANs might just give a nice interface, over the Ansible I'd use to deeply configure the switches.

I still don't know how proxmox HA handles static IP assignments to the VMs. I can see how to use the DNS/ARPA APIs for this, which is awesome: just have a bash script on startup register the IP to the DNS names. Does that mean I need low TTLs on the local hostnames?

I'm still trying to figure out how to do load balancing over multiple VMs on Proxmox as well. As far as I know, I have to run my own nginx/haproxy VM to load balance across known IPs. But, if the IPs are changing... Or the hostname DNS updates (nginx has had issues with this)...

I know I know, the L3 switches will give much better performance and lower latency (e.g. Ceph). I also know I could go the OpenFlow SDN route in hardware; but, it's a bit over my head and the LOE to sit and learn it all I think is overkill for their network.

This is a school of volunteers, and the next person to come in and support this is going to have to use it - usually just us fathers that are technical. having a point-n-click management interface to all of those VLANs with SDN within Proxmox might be the key.

---

My only thought/concern is: how does Proxmox SDN play into Gateways/DMZs/NATs? Do I need an external Gateway per route?

 

[spirit]

I apologize for the newbie questions, as I am trying to wrap my head around Proxmox's SDN vs the rack I am setting up for a school. I would like to list out the basic hardware, and the planned isolation and L3 switches.

• There will be 5x hosts running Proxmox, all fairly equal hardware.
• Ceph will be the backend, across all 5 nodes (OSDs on each node).
• Each host has dual 10G fiber, and dual 1G copper.
• 2x Brocade Enterprise L3 switches
  • Stacked and setup for fail-over/mirror/LAG across the switches.
  • Each host will have a single 10G and a single 1G to one switch, and a LAG setup to the other switch with another 10G and 1G
  • The concept is, if I reboot a switch, they are still up and running on the other switch.
  • 4x VLANs each: Public + Ceph-data over 10G LAG; Corosync + Management over 1G LAG

The requirements for the school are:

• A few backend VMs for the teachers, mostly shared VLAN networking is fine.
• Isolated VLAN management of hosts, switches, etc.
• One "public-facing" app on a static IP (that I am not happy/fully trust).

It's that last one that landed me on this SDN idea for Proxmox. Normally, I would setup some sort of DMZ (between two NATs) to isolate the box from the backend private network.

My plan was to carefully create yet another VLAN dedicated to a virtual DMZ I'll create within the switches - with a NAT gateway running within another VM - back to the switch. This just feels dirty though. Especially if they were able to break out of the VM and onto the Proxmox host... I do not have another "server" to run as a separate gateway/NAT box for that DMZ. Hence, why I am planning on that odd combination.

Would Proxmox SDN help provide a solution here?

Especially since I have Enteprise L3 switches at the ready that supports:

• Full layer 3: IPV4 + IPV6 routing, L2/L3/L4 ACL's, VRRP, OSPF, SNMP, sflow, etc
• BGP, VRFs, tunnels, everything - all high-end L3 switching protocols
• OpenFlow in hardware for SDN, including hybrid port mode
• MACSEC on the SFP+ ports for 80gbps of real time L2 AES-128 encryption

In the end, I prefer a simplified interface vs the in-switch ASIC hardware configurations. And it sounds like this Proxmox SDN implementation with OVS to manage the VLANs might just give a nice interface, over the Ansible I'd use to deeply configure the switches.

I still don't know how proxmox HA handles static IP assignments to the VMs. I can see how to use the DNS/ARPA APIs for this, which is awesome: just have a bash script on startup register the IP to the DNS names. Does that mean I need low TTLs on the local hostnames?

I'm still trying to figure out how to do load balancing over multiple VMs on Proxmox as well. As far as I know, I have to run my own nginx/haproxy VM to load balance across known IPs. But, if the IPs are changing... Or the hostname DNS updates (nginx has had issues with this)...

I know I know, the L3 switches will give much better performance and lower latency (e.g. Ceph). I also know I could go the OpenFlow SDN route in hardware; but, it's a bit over my head and the LOE to sit and learn it all I think is overkill for their network.

This is a school of volunteers, and the next person to come in and support this is going to have to use it - usually just us fathers that are technical. having a point-n-click management interface to all of those VLANs with SDN within Proxmox might be the key.

---

My only thought/concern is: how does Proxmox SDN play into Gateways/DMZs/NATs? Do I need an external Gateway per route?

Hi,
currently gateway/SNAT at hypervisor side could be done only with bgp-evpn sdn plugin. (or if your switch/router support it, you can use them as gateway/at exit too).

(openflow sdn is not implemented, only bgp-evpn)

The roadmap is to add some kind of gateway/nat vm support for vlan plugin or other l2 plugin, or other central service like dhcp/nat1:1/vpn/loadbalancer, but it'll not be ready doon.

About ip assignments to the vms/CT, is not yet implemented, but it should be ready soon. (I hope for proxmox7). it'll be possible to use internal ipam, or other external ipam (phpipam,netbox current) to manage auto ip address assignment, and for dns, powerdns api is currently available for ip registration. (ipam/dns use plugin, so it's possible to easily implement new plugin if needed)

Personnaly, if you want to do simple vlans and don't have physical gateway, I'll go to a simple pair a vm as gateway (pfsense or other). Until you need to route of lot of bandwidth or pps, vms are fine.