proxmox 6.2 sdn beta test

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
Code:
log syslog informational
ip forwarding
ipv6 forwarding
frr defaults datacenter
service integrated-vtysh-config
hostname pve02
!
!
vrf vrf_public
 vni 10000
exit-vrf
!
router bgp 65000
 bgp router-id 192.168.10.2
 no bgp default ipv4-unicast
 coalesce-time 1000
 bgp network import-check
 no bgp ebgp-requires-policy
 neighbor BGP peer-group
 neighbor BGP remote-as external
 neighbor BGP bfd
 neighbor 192.168.10.250 peer-group BGP
 neighbor VTEP peer-group
 neighbor VTEP remote-as 65000
 neighbor VTEP bfd
 neighbor 192.168.128.1 peer-group VTEP
 neighbor 192.168.128.3 peer-group VTEP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family
 !
 address-family ipv6 unicast
  import vrf vrf_public
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor VTEP activate
  advertise-all-vni
 exit-address-family
!
router bgp 65000 vrf vrf_public
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  default-originate ipv4
  default-originate ipv6
 exit-address-family
!
line vty

can your tri to edit frr.conf, and add network X.X.X.X/X in

Code:
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family

Code:
 address-family ipv4 unicast
  network X.X.X.X/X
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family

where X.X.X.X/X is the subnet of your evpn. (ex: "network 192.168.0.0/24")
then restart frr service.


because, I'm not sure than the routes coming from evpn (with the "import vrf_public"), are automaticaly announced.
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
another thing to try, instead "network X.X.X.X/X", try to add "redistribute connected"

Code:
 address-family ipv4 unicast
  redistribute connected
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  import vrf vrf_public
 exit-address-family
 

forsytheda

New Member
Mar 2, 2020
4
0
1
22
Neither of those seemed to work in getting my vm to connected to anything outside of the zone. I am seeing the networks from my router in the route table on pve02, and I see the networks for pve02 in the router, but they are unable to connect. I also am unable to ping the vnet gateway from the proxmox host itself, so that may be there the problem lies.
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
I am seeing the networks from my router in the route table on pve02, and
do you see networks that you want to reach from your vm ?

I see the networks for pve02 in the router,
do you mean the evpn network subnets ? if yes, that's ok the the bgp part ? (is it without "redistributed connected" or "network .X.X.X.X" ?)

can you send the result on pve2 of

Code:
#vtysh
   -->show ip route

but they are unable to connect.
can you try to ping from your vm to outside,

the do a "tcpdump -i vrf_public" to see traffic inside the vrf.

can you try to do it on pve2, to see if traffic is coming from the vm node.

tcpdump "tcpdump -i ethX" on pve2 (where ethx is the interface where the traffic to opense is going through)



I also am unable to ping the vnet gateway from the proxmox host itself, so that may be there the problem lies.
this is expected, the evpn network is in a different vrf. only the exit node have a leak between the vrf to able to route between them.
and the vnet gateway is really special, as it's an anycast gateway. (same ip can be on differents hosts). So it's only respond to the vm directly connected.
 
Last edited:

Nascire

Member
Mar 24, 2012
14
0
21
Wien, Austria
I think I found two "bugs":
SDN itself - Unable to remove Subnet if Gateway specified (makes kind of sense, but the error message is therefore misleading)

PHPIPAM - Unable to specify Gateway - digging makes it seem that the "1" for is_gateway cannot be translated/is not transmitted
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
I think I found two "bugs":
thanks for the repot
SDN itself - Unable to remove Subnet if Gateway specified (makes kind of sense, but the error message is therefore misleading)
I'll try to autoremove gateway from ipam, on subnet removal, if gateway is the last ip from subnet.

PHPIPAM - Unable to specify Gateway - digging makes it seem that the "1" for is_gateway cannot be translated/is not transmitted
can you send me content /etc/pve/sdn/zones.cfg , /etc/pve/sdn/vnets.cfg, /etc/pve/sdn/subnets.cfg , /etc/pve/sdn/ipams.cfg ?
 

Nascire

Member
Mar 24, 2012
14
0
21
Wien, Austria
can you send me content /etc/pve/sdn/zones.cfg , /etc/pve/sdn/vnets.cfg, /etc/pve/sdn/subnets.cfg , /etc/pve/sdn/ipams.cfg ?

Yeah, no problem, but I don´t think that I´ve done anything wrong.
I edited the /usr/share/perl5/PVE/Network/SDN/Ipams/PhpIpamPlugin.pm and added the parameters specified to the error message (line 148 - die "ipam .....")
All parameters showed up as expected, but is_gateway was empty - handing the same call with Postman, the subnet creation and also the addition of the gateway where successful.

On line 147 in /usr/share/perl5/PVE/Network/SDN/SubnetPlugin.pm, from my PoV, the call seems correct, but maybe I´m wrong.

Bash:
root@pve01:~# cat /etc/pve/sdn/*.cfg
phpipam: ipam
        section 3
        token redacted
        url https://ipam.home.xxxxx.site/api/proxmox

pve: pve

subnet: Trunk-192.168.2.0-24
        vnet V0002

subnet: Trunk-192.168.3.0-24
        vnet V0003

subnet: Trunk-172.20.1.0-24
        vnet V0090

vnet: V0002
        zone Trunk
        alias redacted
        tag 2

vnet: V0003
        zone Trunk
        alias redacted
        tag 3

vnet: V0090
        zone Trunk
        alias redacted
        tag 90

vlan: Trunk
        bridge vmbr0
        ipam ipam
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
All parameters showed up as expected, but is_gateway was empty - handing the same call with Postman, the subnet creation and also the addition of the gateway where successful.

I think you are right

n line 147 in /usr/share/perl5/PVE/Network/SDN/SubnetPlugin.pm, from my PoV, the call seems correct, but maybe I´m wrong.
yes, but I totally forget to add $is_gateway in add_ip in Subnets.pm . (I think it was lost during some merge)


Don't have tested it yet, but this should works
Code:
diff --git a/PVE/Network/SDN/Subnets.pm b/PVE/Network/SDN/Subnets.pm
index 46d9830..0231822 100644
--- a/PVE/Network/SDN/Subnets.pm
+++ b/PVE/Network/SDN/Subnets.pm
@@ -232,7 +232,7 @@ sub next_free_ip {
 }
 
 sub add_ip {
-    my ($zone, $subnetid, $subnet, $ip, $hostname, $mac, $description) = @_;
+    my ($zone, $subnetid, $subnet, $ip, $hostname, $mac, $description, $is_gateway) = @_;
 
     return if !$subnet || !$ip; 
 
@@ -259,7 +259,7 @@ sub add_ip {
        my $plugin = PVE::Network::SDN::Ipams::Plugin->lookup($plugin_config->{type});
 
        eval {
-           $plugin->add_ip($plugin_config, $subnetid, $subnet, $ip, $hostname, $mac, $description);
+           $plugin->add_ip($plugin_config, $subnetid, $subnet, $ip, $hostname, $mac, $description, $is_gateway);
        };
        die $@ if $@;
     }

I don't have time to check this until friday, I'll keep you in touch
 

Nascire

Member
Mar 24, 2012
14
0
21
Wien, Austria
Don't have tested it yet, but this should works


I don't have time to check this until friday, I'll keep you in touch

That did the trick :)

First try was without deleting the IP from IPAM, resulting in error 409 object already exists.
Afterwards I deleted it from IPAM and tried again, finally got it working.

In case of the gateway IP, maybe it would be good to update the entry in IPAM (or at least, ask if you want to)?
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
That did the trick :)

First try was without deleting the IP from IPAM, resulting in error 409 object already exists.
Afterwards I deleted it from IPAM and tried again, finally got it working.
ok, great! I'll send a patch next week to pve-devel mailing list.

In case of the gateway IP, maybe it would be good to update the entry in IPAM (or at least, ask if you want to)?
Yes, I will check that.

Thanks again for testing !
 

ph0x

Active Member
Jul 5, 2020
917
144
43
/dev/null
I setup a virtual test cluster yesterday since I'm very interested in this sdn feature and for now it looks pretty promising.
Can you point me towards some resources where I can read about best practices? For example, I have setup three zones (as in three "data centers") and several vtnets inside the zones (as in traditional VLANs), is this the way to think of it?
Am I right that I would then need to setup gateway VMs to route between those vtnets and zones?
One thing that I noticed: the "Alias" tag of the vtnets, which is defined in the sdn section does not get propagated to the zone sections (zone->content) of the participating hosts, that field is just empty.
 
Last edited:

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
I setup a virtual test cluster yesterday since I'm very interested in this sdn feature and for now it looks pretty promising.
Can you point me towards some resources where I can read about best practices?
The current ressources are in
https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

but I think I should add more examples

For example, I have setup three zones (as in three "data centers") and several vtnets inside the zones (as in traditional VLANs), is this the way to think of it?
mmm, no. A zone is more a security zone, than a physical zone. Vnets from a zone should not be able to communicate with vlans in other zone.
It's also possible to give permission to a user on a specific zone, and He will see only vnets from this zone.
For QinQ for example, you have a main vlan on the zone, and sub vlans on vnets.
for bgp-evpn (routage vxlan), a vxlan from a zone can be routed to a vxlan on other zone.
For vlans, a gateway in a zone should not be able to route vlans from others zones


Am I right that I would then need to setup gateway VMs to route between those vtnets and zones?
yes, the gateway is not yet implemented in proxmox. (but it should come soon, currently in discussion).
so you need to implement your own gateway (vm or physical, what you want).

Basicaly, for vlan, it's the same than define them locally on each server vmbrX, but it's done once at datacenter level


One thing that I noticed: the "Alias" tag of the vtnets, which is defined in the sdn section does not get propagated to the zone sections (zone->content) of the participating hosts, that field is just empty.
The alias is really just an alias/description (no characters limit) of the vnet (limited to 8 characters). I'll be display later in the vm nic gui when you choose the bridge.
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
@Nascire I have send patch to pve-devel mailing list to fix the is_gateway, and I also allow to reuse the gateway ip if it's already in ipam with "is_gateway" flag. Like this, it's possible to redeclare subnets on multiple proxmox clusters sharing the same ipam.

I'll should be available soon in new package version.

Thanks again for reporting this bug.
 

woelfl

New Member
Jun 7, 2021
1
0
1
36
Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:
Screenshot at Jun 07 13-48-14.png

Bug?
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:
View attachment 26617

Bug?
Hi, mmm, I think I have forgot something in the last version. (I don't have worked too much in permissions, I still need to add pool permissions too)
I'll check that today or tomorrow, thanks for report this.
 
  • Like
Reactions: woelfl

ph0x

Active Member
Jul 5, 2020
917
144
43
/dev/null
mmm, no. A zone is more a security zone, than a physical zone. Vnets from a zone should not be able to communicate with vlans in other zone.
Yeah, understood. That was my intention, already assuming that "datacenter" might bring in the wrong notion. :)

yes, the gateway is not yet implemented in proxmox. (but it should come soon, currently in discussion).
so you need to implement your own gateway (vm or physical, what you want).
Check.

The alias is really just an alias/description (no characters limit) of the vnet (limited to 8 characters). I'll be display later in the vm nic gui when you choose the bridge.
Check so far, but when I view a zone under a host, there already is an uneditable column for the Alias, which - I assume - should be filled with the Alias definition of the SDN section as well.
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
Check so far, but when I view a zone under a host, there already is an uneditable column for the Alias, which - I assume - should be filled with the Alias definition of the SDN section as well.
oh, you're right. I forgot to add the content. I'll fix that asap, thanks for reporting !
 
  • Like
Reactions: ph0x

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
Hi guys. Did someone successfully applied a permission to a zone yet? I tried via gui and api, but getting stuck with:

path: invalid ACL path '/sdn/zones/<zonename>'

Version: 6.4-8, see:
View attachment 26617

Bug?
I have send a fix the pve-devel mailing.
It seem to have breaked recentyly

Code:
diff --git a/src/PVE/AccessControl.pm b/src/PVE/AccessControl.pm
index 888875e..4754c5c 100644
--- a/src/PVE/AccessControl.pm
+++ b/src/PVE/AccessControl.pm
@@ -945,6 +945,7 @@ sub check_path {
        |/pool
        |/pool/[[:alnum:]\.\-\_]+
        |/sdn
+        |/sdn/zones/[[:alnum:]\.\-\_]+
        |/storage
        |/storage/[[:alnum:]\.\-\_]+
        |/vms

It's just missing a path check in accesscontrol, you can try to add the missing line manually in /usr/share/perl5/PVE/AccessControl.pm ,
and restart pveproxy daemon.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!