proxmox 7.0 sdn beta test

spirit · Aug 7, 2020

nadeu said:
Can I use this together with DPDK?

I dont have added support to dpdk yet. But technically, it could be possible to do a plugin. But it still miss support for vhost-user for qemu. Ill try to add it later, After first stable release. i m currently working on IP management through ipam && dns registration.

nadeu · Aug 10, 2020

spirit said:
I dont have added support to dpdk yet. But technically, it could be possible to do a plugin. But it still miss support for vhost-user for qemu. Ill try to add it later, After first stable release. i m currently working on IP management through ipam && dns registration.

Added you on LinkedIn, can continue this thread on it.

jeffrey404 · Aug 14, 2020

So I am having the issue when using SDN in my proxmox setup. I am trying a simple vlan setup and I am getting the following error when trying to start a VM:

Code:

/dev/rbd9
bridge 'zbxint' does not exist
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
TASK ERROR: start failed: QEMU exited with code 1

I have setup the zone and vnet in the gui according to the instructions. Here is the vnets.cfg and zones.cfg

Code:

root@hv01:~# cat /etc/pve/sdn/vnets.cfg
vnet: zbxpri01
        tag 585
        zone zbxpri01
        alias 10.153.6.0/24
        ipv4 10.153.6.0/24
        vlanaware 1

vnet: zbxint
        tag 575
        zone zbxpri01
        alias 10.153.7.0/24
        ipv4 10.153.7.0/24
        vlanaware 1

root@hv01:~# cat /etc/pve/sdn/zones.cfg
vlan: zbxpri01
        bridge vmbr1

root@hv01:~#

Here is my network config on the host that is throwing the error:

Code:

root@hv01:/etc/network# cat interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto enp67s0f0
iface enp67s0f0 inet manual

auto enp67s0f1
iface enp67s0f1 inet manual
        mtu 9000

auto enp68s0f0
iface enp68s0f0 inet manual

auto enp68s0f1
iface enp68s0f1 inet manual
        mtu 9000

auto cephnet
iface cephnet inet static
        address xxx.xxx.xxx.xxx
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        ovs_mtu 9000
        ovs_options tag=300

auto bond0
iface bond0 inet manual
        ovs_bonds enp67s0f1 enp68s0f1
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_mtu 9000
        ovs_options bond_mode=balance-slb lacp=active

auto bond1
iface bond1 inet manual
        ovs_bonds enp67s0f0 enp68s0f0
        ovs_type OVSBond
        ovs_bridge vmbr2
        ovs_options bond_mode=balance-slb lacp=active

auto vmbr0
iface vmbr0 inet static
        address xxx.xxx.xxx.xxx
        gateway xxx.xxx.xxx.xxx
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 cephnet
        ovs_mtu 9000

auto vmbr2
iface vmbr2 inet manual
        ovs_type OVSBridge
        ovs_ports bond1

I feel like it is something simple that I am missing. Thanks in advanced.

spirit · Aug 15, 2020

jeffrey404 said:
I feel like it is something simple that I am missing. Thanks in advanced.

Hi, thanks for testing !

it's missing "source /etc/network/interfaces.d/*" in /etc/network/interfaces

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_local_deployment_monitoring

it's needed to load generated configuration in /etc/network/interfaces.d/sdn.

(I'll try to add a warning on reload soon, as you are not the first to ask the question

jeffrey404 · Aug 15, 2020

spirit said:
Hi, thanks for testing !

it's missing "source /etc/network/interfaces.d/*" in /etc/network/interfaces

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_local_deployment_monitoring

it's needed to load generated configuration in /etc/network/interfaces.d/sdn.

(I'll try to add a warning on reload soon, as you are not the first to ask the question

Yup, that was the problem. Apparently, I need to learn to read... haha... Thanks for your help.

spirit · Aug 16, 2020

jeffrey404 said:
Yup, that was the problem. Apparently, I need to learn to read... haha... Thanks for your help.

No problem. Note that i m currently working on IP management, so the IP field on vnet will change soon. (If you use a routed setup with simple plugin with vnet IP as gateway ), so don't use it yet in production. (The other parts of current sdn implementation shouldn't change)

jlebherz · Sep 14, 2020

hey there,

I get the following error by using vxlan with vlanAware

error: vxlan_vx99: `bridge-access` attribute is mandatory when vxlan device (vxlan_vx99) is part of vlan aware bridge (vx99)
warning: vx99: `bridge-access` attribute is mandatory when vxlan device (vxlan_vx99) is part of vlan aware bridge (vx99)
warning: vx99: apply bridge settings: vx99: errors applying port settings

/etc/network/interfaces.d/sdn

Code:

auto vx99
iface vx99
        bridge_ports vxlan_vx99
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vxlan_vx99
iface vxlan_vx99
        vxlan-id 99
        vxlan_remoteip 172.30.31.3
        mtu 9000

if I add bridge-access 1 to the vxlan configuration everything is great.

working config

Code:

auto vx99
iface vx99
        bridge_ports vxlan_vx99
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vxlan_vx99
iface vxlan_vx99
        vxlan-id 99
        vxlan_remoteip 172.30.31.3
        bridge-access 1
        mtu 9000

I know tahts not an ideal solution editing the sdn file...

Has anyone of you the same problem?

thanks!

Versions:
ifupdown2:3.0.0-1+pve2
pve-manager/6.2-11/22fb4983 (running kernel: 5.4.60-1-pve)
libpve-network-perl 0.4-6

spirit · Sep 15, 2020

@jlebherz

Thanks for the report!

I'll look inside ifupdown2, I known that they limited vlan inside vxlan, because last kernel version have vlan-to-vxlan mapping.
But technically, it's possible to do vlans inside vxlan tunnel.

I'll keep you in touch today.

f I add bridge-access 1 to the vxlan configuration everything is great.

ok thanks.
Do you have tried to add "vlan 1" in the gui for this vnet ? (+vlanaware)

fhloston · Sep 18, 2020

I can confirm that bridge-access 1 makes this work for vlanaware 1 \o/

but bridge-access 1 obviously makes only vlan=1 work...

spirit · Sep 18, 2020

fhloston said:
I can confirm that bridge-access 1 makes this work for vlanaware 1 \o/

but bridge-access 1 obviously makes only vlan=1 work...

Hi, I known how to fix it, I'll send a patch soon.

I have 2 possiblity:

- Allow vlan inside the vxlan tunnel. (I need to patch ifupdown2, It's an arbitrary limitation)
- Map vlan to vxlan with new kernel feature. (vlanid 1000 become vxlanid 1000 when going to vxlan interface, and rettaged vlanid1000 on exist at the other side))

I'll see what is the best way.

spirit · Sep 21, 2020

@fhloston

I have done a lot of tests, it's working fine with vlan inside vxlan tunnel
I'll send a new version of ifupdown2 soon.

if you want to hack it, you can edit

/usr/share/ifupdown2/addons/bridge.py

and comment

Code:

#        if not isbridge and (bportifaceobj.link_kind & ifaceLinkKind.VXLAN and not bportifaceobj.link_privflags & ifaceLinkPrivFlags.SINGLE_VXLAN):
#            if not vids or not pvid or len(vids) > 1 or vids[0] != pvid:
#                self._error_vxlan_in_vlan_aware_br(bportifaceobj,
#                                                   bportifaceobj.upperifaces[0])
#                return

it's should works fine now

spirit · Sep 24, 2020

@fhloston

here a patch ifupdown2 package, it should allow vlan inside vxlan

http://odisoweb1.odiso.net/ifupdown2_3.0.0-1+pve2_all.deb

Please test!

(I have sent patch to pve-devel mailing list, so a new version with the patch should be available soon)

fhloston · Sep 25, 2020

@spirit

indeed, with this patched version of ifupdown2 it works as expected!

\o/ Thanks!

spirit · Sep 25, 2020

fhloston said:
@spirit

indeed, with this patched version of ifupdown2 it works as expected!

\o/ Thanks!

official deb should be available soon in proxmox repos

guletz · Sep 28, 2020

Hi to all,

I have use the the setup from here:

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html


auto vmbr0
iface vmbr0 inet static
        address 192.168.x.108/24
        gateway 192.168.x.205
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
source /etc/network/interfaces.d/*

... by the way "source ... was kill me, because I did not see it, maby a BOLD note about?"


cat /etc/pve/sdn/zones.cfg 
vxlan: myvxlan
        peers 192.168.x.101, 192.168.x.102, 192.168.x.103, 192.168.x.104, 192.168.x.5, 192.168.x.7, 192.168.x.108
        mtu 1450


cat /etc/pve/sdn/vnets.cfg 
vnet: vnet1
        tag 100001
        zone myvxlan
        alias vxlan01
        vlanaware 0

Then I add a new interface in bridge "vnet1" for 2 VM, with IP address in the same network(like 192.168.4.x/24, mtu 1450). But I can not ping from one of such VM to the other(like ping 192.168.4.x1), and also the arp -an did not show any entry for the other IP!

Thx. in advance

spirit · Sep 29, 2020

... by the way "source ... was kill me, because I did not see it, maby a BOLD note about?"

I think I'll look to auto add it in /etc/network/interfaces.

Then I add a new interface in bridge "vnet1" for 2 VM, with IP address in the same network(like 192.168.4.x/24, mtu 1450). But I can not ping from one of such VM to the other(like ping 192.168.4.x1), and also the arp -an did not show any entry for the other IP!

Thx. in advance

Can you provide result of

Code:

# ip -d a
# brctl show
# ifreload -a -d

of both source && destination nodes ?

fhloston · Sep 29, 2020

spirit said:
official deb should be available soon in proxmox repos

I guess ifupdown2:3.0.0-1+pve3 contains the fix for vlans on vxlan?

fhloston · Sep 29, 2020

guletz said:
I have use the the setup from here:

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html
(...)

Then I add a new interface in bridge "vnet1" for 2 VM, with IP address in the same network(like 192.168.4.x/24, mtu 1450). But I can not ping from one of such VM to the other(like ping 192.168.4.x1), and also the arp -an did not show any entry for the other IP!

Did you actually apply that config?

pvesh set /cluster/sdn
ifreload -a

spirit · Sep 29, 2020

fhloston said:
I guess ifupdown2:3.0.0-1+pve3 contains the fix for vlans on vxlan?

yes !

spirit · Sep 29, 2020

fhloston said:
Did you actually apply that config?
pvesh set /cluster/sdn ifreload -a

yes, indeed you need to apply the config.
"pvesh set /cluster/sdn" is enough, it's reloading config on all nodes.
or you can use the "apply" button on the sdn gui.

I think that currently, the vnetx are display in vm nic form even if they are not yet applied. (but vm shouldn't start as bridge is not yet available).
I have changed that in coming version, where you'll see only applied config in vm nic form

proxmox 7.0 sdn beta test

Distinguished Member

New Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Distinguished Member

Distinguished Member

Member

Distinguished Member

Famous Member

Distinguished Member

Member

Member

Distinguished Member

Distinguished Member