proxmox 7.0 sdn beta test

pieteras.meyer · May 18, 2020

Hi Spirit

Also seems that vnet interface cannot be hotplug added or removed on a running VM get the below error in the GUI

Parameter verification failed. (400)
net2: hotplug problem - error on hot-unplugging device 'net2

spirit · May 18, 2020

pieteras.meyer said:

Code:

()
ovs vmbr0 is not supported by qinq at /usr/share/perl5/PVE/Network/SDN/Zones/QinQPlugin.pm line 94.
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
TASK ERROR: start failed: QEMU exited with code 1

ovs is not supported by qinq plugin, sorry. (I have only implemented it over linux vlan-aware brigde,
as I don't have physical switch to test it.
Do you need qinq with ovs ? if yes, help is welcomed for testing it.

I known that you can use vlan_mode=dot1q-tunnel with ovs, but I'm not sure where/how to define both vlan tags.
(from my understanding, the customer tag is already exist, so tagged from inside the vm)

I see in the forum your previous post, so maybe do you use it already ?

Also, about qinq, I'm not sure how/where the handle the customer vlan.
Currently service vlan is defined on the zone, then you defined customer vlan on vnets .
I'm don't have if we wan't to allow final user to defined the customer vlan themself? (in the gui ? from the nic).
Seem that we can have a lot of combinations, I'm really not sure how to handle that.

spirit · May 18, 2020

pieteras.meyer said:
Hi Spirit

Also seems that vnet interface cannot be hotplug added or removed on a running VM get the below error in the GUI
Parameter verification failed. (400)
net2: hotplug problem - error on hot-unplugging device 'net2

do you try to hotplug/unplug a nic on a vnet? (I mean, not changing the vnet, but adding a new nic ?
if yes, with which zone plugin ? vlan + ovs ?

Mike Lupe · May 18, 2020

spirit said:
@Mike Lupe

I have done a lot of test, I can't reproduce on my side :/
What is your vm guest os ? virtio nic ? (if yes, can you try with e1000 ?)
Can you also try to use vm nic disconnect checkbox twice ?

Hi @spirit , OS is Debian 9.x

Unfortunately it's still not working, even with e1000

1. Changed nic (live) from virtio to E1000
# ping ok
2. Enabled Firewall on e1000 nic
# ping ok
3. Applyied SDN (without other changes)
# ping not ok
4. Rebooted VM
# ping ok
5. Again applied SDN (without other changes)
# ping not ok
6. Disabled FW on e1000 nic
# ping ok

7. Re-enabled FW on e1000 nic
# ping ok
8. Again applied SDN (without other changes)
# ping not ok
9. "Disconnect" on e1000 nic
# ping not ok
10. Disabled "Disconnect" on e1000 nic (so re-enabled nic)
# ping not ok
11. "Disconnect" on e1000 nic second time
# ping not ok
12. Disabled "Disconnect" on e1000 nic (so re-enabled nic)
# ping not ok

13. Disabled FW on e1000 nic
# ping ok

FYI I don't really need the local PVE firewall, it was a pure coincidence the FW was activated on that VM. But for the completeness sake maybe we get around this issue

Mike Lupe · May 18, 2020

pieteras.meyer said:
Hi Spirit

... then the reload seem to work fine, I dont make use of firewall so can confirm the bug mike is seeing,

Hi @pieteras.meyer ,

did you see the bug only now , because otherwise you don't use the FW, or was it a typo and you "can't" confirm it?

spirit · May 18, 2020

Mike Lupe said:
Hi @spirit , OS is Debian 9.x

Unfortunately it's still not working, even with e1000

1. Changed nic (live) from virtio to E1000
# ping ok
2. Enabled Firewall on e1000 nic
# ping ok
3. Applyied SDN (without other changes)
# ping not ok
4. Rebooted VM
# ping ok
5. Again applied SDN (without other changes)
# ping not ok
6. Disabled FW on e1000 nic
# ping ok

7. Re-enabled FW on e1000 nic
# ping ok
8. Again applied SDN (without other changes)
# ping not ok
9. "Disconnect" on e1000 nic
# ping not ok
10. Disabled "Disconnect" on e1000 nic (so re-enabled nic)
# ping not ok
11. "Disconnect" on e1000 nic second time
# ping not ok
12. Disabled "Disconnect" on e1000 nic (so re-enabled nic)
# ping not ok

13. Disabled FW on e1000 nic
# ping ok

FYI I don't really need the local PVE firewall, it was a pure coincidence the FW was activated on that VM. But for the completeness sake maybe we get around this issue

does a simple "ifreload -a" also break vm network ?
if yes, can you send me output of "ifreload -a -d" ?

Mike Lupe · May 18, 2020

spirit said:
does a simple "ifreload -a" also break vm network ?
if yes, can you send me output of "ifreload -a -d" ?

Exactly the same happens with "ifreload -a".

Debuglog is prepared, I'll send you a "Laissez nous un message" on your website for further email contact, ok?

edit: In the meantime, here some "not founds" as a log extract:

Code:

info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module ppp not loaded (module init failed: no /usr/bin/pon found)
info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)
[...]
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

pieteras.meyer · May 18, 2020

spirit said:
ovs is not supported by qinq plugin, sorry. (I have only implemented it over linux vlan-aware brigde,
as I don't have physical switch to test it.
Do you need qinq with ovs ? if yes, help is welcomed for testing it.

I known that you can use vlan_mode=dot1q-tunnel with ovs, but I'm not sure where/how to define both vlan tags.

I see in the forum your previous post, so maybe do you use it already ?

Hi Spirit

I will be happy to help test this and get it implemented

Yes We only make use of OVS and we do make use of QinQ , currently I have added the patch from David herselman from the below post

https://forum.proxmox.com/threads/proxmox-5-0-and-ovs-with-dot1q-tunnel.34090/

https://bugzilla.proxmox.com/show_bug.cgi?id=1350

I get this done by adding linux vlan to a OVS port and then adding a new OVS bridge and then adding the linux vlan as a port to the new bridge,

you get 2 types of Stags the default 802.1Q (0x8100) or 802.1ad (0x88a8) the 0x88a8 is the service provider vlan tag and not all switches support this,

https://developers.redhat.com/blog/2017/06/06/open-vswitch-overview-of-802-1ad-qinq-support/
https://developers.redhat.com/blog/2017/06/27/open-vswitch-qinq-performance/

I currently make use of normal 0x8100 tags

Code:

# Add qinq vlan for customer
ip link add link eth0 name eth0.4048 mtu 9000 type vlan id 4048
ip link set eth0.4048 up
ovs-vsctl add-br vmbr4048
ovs-vsctl add-port vmbr4048 eth0.4048
add info to from /etc/network/interfaces
#eth0.4048 - aurum

allow-vmbr4048 eth0.4048
iface eth0.4048 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4048
        mtu 9000
#eth0.4048 - aurum

auto vmbr4048
iface vmbr4048 inet manual
        ovs_type OVSBridge
        ovs_ports eth0.4048
        mtu 9000
# aurum stag 4048

ovs-vsctl show
ovs-vsctl list-br
ovs-vsctl list-ports vmbr4048
######################
#remove qinq vlan for customer
ovs-vsctl del-port vmbr4048 eth0.4048
ovs-vsctl del-br vmbr4048
ip link set eth0.4048 down
ip link delete eth0.4048
remove from /etc/network/interfaces
see above fields that need to be removed
#

Code:

# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
# https://forum.proxmox.com/threads/pve5-fix-those-predictable-network-interface-names.37210/
# https://forum.proxmox.com/threads/ifconfig-not-included.49664/
#eth0 - 1_Ex0/13 - 10G
#eth1 - 2_Ex0/13 - 10G
#eth2 - 1_Gi0/13 - 1G
#eth3 - 2_Gi0/13 - 1G
#
auto lo
iface lo inet loopback
        pre-up ifconfig eth0 mtu 9000
        pre-up ifconfig eth1 mtu 9000
        pre-up ifconfig eth2 mtu 9000
        pre-up ifconfig eth3 mtu 9000
# Set mtu on physical interfaces
#######################################################
#eth0 - 1_Ex0/13 - 10G

allow-vmbr0 eth0
iface eth0 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0
        mtu 9000
#1_Ex0/13 - 10G

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports eth0 vlan1
        mtu 9000
#guest network 10G

#######################################################
#######################################################

#eth1 - 2_Ex0/13 - 10G # storage network

allow-vmbr18 eth1
iface eth1 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr18
        mtu 9000
#2_Ex0/13 - 10G

auto vmbr18
iface vmbr18 inet manual
        ovs_type OVSBridge
        ovs_ports eth1 vlan18 vlan20 vlan21
        mtu 9000
# storage network

#######################################################
#######################################################
#eth2 - 1_Gi0/13 - 1G # corosync network b

allow-vmbr17 eth2
iface eth2 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr17
        mtu 9000
#1_Gi0/13 - 1G

# corosync network b
auto vmbr17
iface vmbr17 inet manual
        ovs_type OVSBridge
        ovs_ports eth2 vlan17
        mtu 9000
# corosync network b

#######################################################
#######################################################

#eth3 2_Gi0/13 - 1G
#eth3.4048 - aurum

allow-vmbr1 eth3
iface eth3 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr1
        mtu 9000
#2_Gi0/13 - 1G

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports eth3 vlan16
        mtu 9000
# corosync network a

allow-vmbr4048 eth3.4048
iface eth3.4048 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4048
        mtu 9000
#eth3.4048 - aurum

auto vmbr4048
iface vmbr4048 inet manual
        ovs_type OVSBridge
        ovs_ports eth3.4048
        mtu 9000
# aurum stag 4048

#eth3.4057 - broll_eoh

allow-vmbr4057 eth3.4057
iface eth3.4057 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4057
        mtu 9000
#eth3.4057 - broll_eoh

auto vmbr4057
iface vmbr4057 inet manual
        ovs_type OVSBridge
        ovs_ports eth3.4057
        mtu 9000
# broll stag 4057

#eth3.4059 - comsol

allow-vmbr4059 eth3.4059
iface eth3.4059 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4059
        mtu 9000
#eth3.4059 - comsol

auto vmbr4059
iface vmbr4059 inet manual
        ovs_type OVSBridge
        ovs_ports eth3.4059
        mtu 9000
# comsol stag 4059

#eth3.4065 - bitco

allow-vmbr4065 eth3.4065
iface eth3.4065 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4065
        mtu 9000
#eth3.4065 - bitco

auto vmbr4065
iface vmbr4065 inet manual
        ovs_type OVSBridge
        ovs_ports eth3.4065
        mtu 9000
# bitco stag 4065

#eth3.4067 - liquid

allow-vmbr4067 eth3.4067
iface eth3.4067 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr4067
        mtu 9000
#eth3.4067 - liquid

auto vmbr4067
iface vmbr4067 inet manual
        ovs_type OVSBridge
        ovs_ports eth3.4067
        mtu 9000
# liquid stag 4067

#######################################################
#######################################################

#guest network 10G
allow-vmbr0 vlan1
iface vlan1 inet static
        address  10.254.1.55
        netmask  255.255.255.0
        gateway  10.254.1.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        mtu 9000
#guest network 10G

# corosync network b
allow-vmbr1 vlan16
iface vlan16 inet static
        address  10.255.16.5
        netmask  255.255.255.240
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        ovs_options tag=16
        mtu 9000
# corosync network b

# corosync network a
allow-vmbr17 vlan17
iface vlan17 inet static
        address  10.255.17.5
        netmask  255.255.255.240
        ovs_type OVSIntPort
        ovs_bridge vmbr17
        ovs_options tag=17
        mtu 9000
# corosync network a

# storage network
allow-vmbr18 vlan18
iface vlan18 inet static
        address  10.255.18.5
        netmask  255.255.255.240
        ovs_type OVSIntPort
        ovs_bridge vmbr18
        ovs_options tag=18
        mtu 9000
#storage network

# storage iSCSI 1
allow-vmbr18 vlan20
iface vlan20 inet static
        address  10.255.20.5
        netmask  255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr18
        ovs_options tag=20
        mtu 9000
# storage iSCSI 1

# storage iSCSI 2
allow-vmbr18 vlan21
iface vlan21 inet static
        address  10.255.21.5
        netmask  255.255.255.0
        ovs_type OVSIntPort
        ovs_bridge vmbr18
        ovs_options tag=21
        mtu 9000
# storage iSCSI 2

#######################################################

vlan interface

pieteras.meyer · May 18, 2020

spirit said:
do you try to hotplug/unplug a nic on a vnet? (I mean, not changing the vnet, but adding a new nic ?
if yes, with which zone plugin ? vlan + ovs ?

Hi Spirit
Yes I have adding a new vnet nic to a runnning VM

the Zone was QinQ to linuxbridge vlanaware,

spirit · May 18, 2020

pieteras.meyer said:
Hi Spirit
Yes I have adding a new vnet nic to a runnning VM

the Zone was QinQ to linuxbridge vlanaware,

ok thanks, maybe it's a bug in qinq plugin. (I don't have problem with vlan plugin).
I'll test it, thanks

spirit · May 18, 2020

Mike Lupe said:
Exactly the same happens with "ifreload -a".

Debuglog is prepared, I'll send you a "Laissez nous un message" on your website for further email contact, ok?

you can pm me on the forum, or send me a mail to "aderumier@odiso.com"

edit: In the meantime, here some "not founds" as a log extract:

Code:

info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module ppp not loaded (module init failed: no /usr/bin/pon found)
info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)
[...]
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)
info: module ethtool not loaded (module init failed: /sbin/ethtool: not found)

no problem with this, some ifupdown2 are not loaded because we don't install all binary in proxmox.

spirit · May 18, 2020

pieteras.meyer said:
I get this done by adding linux vlan to a OVS port and then adding a new OVS bridge and then adding the linux vlan as a port to the new bridge,

ok thanks. I was not sure if it was possible to create both tag with ovs directly. So with linux vlan interface is more clear now.

Any special reason to use ovs instead a simple linux interface double tag ? (eth0.X.y --->vmbr)

Mike Lupe · May 18, 2020

spirit said:
you can pm me on the forum, or send me a mail to "aderumier@odiso.com"

Mail with log is sent.

no problem with this, some ifupdown2 are not loaded because we don't install all binary in proxmox.

Ok, thanks for the info.

Mike Lupe · May 18, 2020

@spirit We worked it out, great.

Sorry, it was my bad, having on exactly that node with this specific VM still the older ifupdown2 version from May 16th. It's when you realize there's a .deb.1 file around

With the CT it worked, because on that node where the CT resides, the updated ifupdown2 was already installed.

spirit · May 18, 2020

@pieteras.meyer

I have found multiple bug with qinq plugin (with vlan-aware).
I think I don't have tested it well.
I'll try to fix it for tomorrow, and maybe look at ovs too.

subbsd · May 18, 2020

There are opportunities for SDN debug/verbose ? I created:

1) Zone: 'myvz', Peer Address List: 'x.x.x.x, y.y.y.y' ( x.x.x.x/y.y.y.y - external IPv4 node's address )
2) Vnets: 'test', Zone: 'myvx'

On 'apply' i got: error'

No any vxlan/SDN-related in /var/log/*

Code:

% pveversion
pve-manager/6.2-4/9824574a (running kernel: 5.4.41-1-pve)

pve-kernel-5.4 6.2-2
ifupdown2 2.0.1-1+pve8
ibpve-network-perl 0.4-4

spirit · May 18, 2020

subbsd said:
There are opportunities for SDN debug/verbose ? I created:

1) Zone: 'myvz', Peer Address List: 'x.x.x.x, y.y.y.y' ( x.x.x.x/y.y.y.y - external IPv4 node's address )
2) Vnets: 'test', Zone: 'myvx'

On 'apply' i got: error'

No any vxlan/SDN-related in /var/log/*

Code:

% pveversion pve-manager/6.2-4/9824574a (running kernel: 5.4.41-1-pve)

pve-kernel-5.4 6.2-2
ifupdown2 2.0.1-1+pve8
ibpve-network-perl 0.4-4

Hi,

can you send content of

sdn configuration
---------------------------
/etc/pve/sdn/vnets.cfg
/etc/pve/sdn/zones.cfg

local generated configuration
-------------------------------------------
/etc/network/interfaces
/etc/network/interfaces.d/sdn

can you send output of "ifquery -c -a" ?
the error reporting is mainly based on ifquery.

and also, try a reload in debug mode with "ifreload -a -d" and send the result.

spirit · May 19, 2020

@pieteras.meyer

I have reworked the qinqplugin, it's available here:

Code:

wget http://odisoweb1.odiso.net/libpve-network-perl_0.4-4_all.deb
dpkg -i libpve-network-perl_0.4-4_all.deb
systemctl restart pvedaemon

It's fixed qinq with vlan-aware bridge, and add support for ovs too !
default in 802.1q for service vlan + 802.1q for customer vlan

if you need 802.1ad for service vlan, you can edit
/etc/pve/sdn/zones.cfg

and add "vlan-protocol 802.1ad" option to your qinq zone.

(I'll update gui tomorrow)

So, currently, service vlan are defined on zone, customer vlan is defined in vnets.

Do you have the need to define customer vlan inside the vm ?
or allow user to any vlan tag on the vm nic (like use the zone as bridge, and define tag you want)

pieteras.meyer · May 19, 2020

spirit said:
ok thanks. I was not sure if it was possible to create both tag with ovs directly. So with linux vlan interface is more clear now.

Any special reason to use ovs instead a simple linux interface double tag ? (eth0.X.y --->vmbr)

Hi Spirit

We try and keep as much as possible in OVS, but OVS cannot add 2 layers of STag so that is why I add 1 layer with linux vlan before using the dot1q-tunnel

spirit · May 19, 2020

pieteras.meyer said:
Hi Spirit

We try and keep as much as possible in OVS, but OVS cannot add 2 layers of STag so that is why I add 1 layer with linux vlan before using the dot1q-tunnel

I have done it differently in my plugin, without ethx.Y.

I'm doing it like this:

eth0---->ovs---(dot1q-tunnel port tag)--------linuxvlanawarebrige--- (tag)----vm.

(I could use another ovs switch, but to be cleaner in my code, I currently use a linuxvlanaware bridge)

Like this, later I could allow also tagging inside the vm, instead the vnet.

I have done tcpdump with ovs && linuxbridge, I'm correctly seeing double tagged packets.
(also verify with 802.1ad or 802.1q for service vlan)

proxmox 7.0 sdn beta test

Renowned Member

Distinguished Member

Distinguished Member

Member

Member

Distinguished Member

Member

Renowned Member

Renowned Member

Distinguished Member

Distinguished Member

Distinguished Member

Member

Member

Distinguished Member

Member

Attachments

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member