proxmox 7.0 sdn beta test

[Sebastian Schubert]

Hi there,

I'm a bit courious about relying on ifupdown2 when seeing what happened upstream to ifupdown2 (https://bugs.launchpad.net/ubuntu/+source/ifupdown2/+bug/1872590)

 

[spirit]

Hi there,

I'm a bit courious about relying on ifupdown2 when seeing what happened upstream to ifupdown2 (https://bugs.launchpad.net/ubuntu/+source/ifupdown2/+bug/1872590)

Ubuntu use netplan by default, so this don't make any sense for them to keep ifupdown1 or ifupdown2 or any think related to /etc/network/interfaces. (and netplan is just a wrapper on top of systemd-networkd)


Ifupdown2 is developped by cumulus linux, used in their switch. and main ifupdown2 developper is also the maintainer of debian package
and version 3 has been migrated to testing recently.
https://tracker.debian.org/news/1177218/ifupdown2-300-1-migrated-to-testing/

So no worry here, it'll be available for a long time.

(and it's far more advanced than netplan/systemd-networkd)

 

[t.lamprecht]

I'm a bit courious about relying on ifupdown2 when seeing what happened upstream to ifupdown2 (https://bugs.launchpad.net/ubuntu/+source/ifupdown2/+bug/1872590)

Note that, while yes, an older version was removed from Debian Testing due to it not being ported to python3, the recent ifupdown2 3.0 release was then introduced to testing/unstable Debian repositories again:
https://tracker.debian.org/news/1177218/ifupdown2-300-1-migrated-to-testing/

This one is now updated to python 3 and will stay.

Removing packages temporarily from Debian testing or unstable is a pretty common workflow in Debian, so one needs to always check the context here - which is not always obvious, like it was the case for the Ubuntu Tracker you linked.

Anyway, as spirit said: ifupdown2 has its Debian maintainer also as developer, which is really nice.
Further, we package ifupdown2 ourself to have more recent versions (with all the good bug and feature fixes @spirit does included).

 

[cyruspy]

I've tried this between two test nodes (not part of a cluster) and VXLAN, the traffic seems to pass through, but ping tests reports duplicated answers:



Any hints?

 

[cyruspy]

Given IDs are global, wouldn't make more sense to generate automatically an UUID for the VNET?, "inviting" the user to populate Alias parameter with what makes sense descriptively to the user?.

Also, given VNIs are global, should de interface autogenerate one as proposal?, and the user would be able to change it if needed (to match for example VNIs on an external node?)

 

[spirit]

Given IDs are global, wouldn't make more sense to generate automatically an UUID for the VNET?, "inviting" the user to populate Alias parameter with what makes sense descriptively to the user?.

Also, given VNIs are global, should de interface autogenerate one as proposal?, and the user would be able to change it if needed (to match for example VNIs on an external node?)

Hi, I can't use uuid where, because of limitation of linux interfaces name lenght. (14caracters), and I'm currently limiting vnet to 8 caracters, because I'm also use some internal mapping between vnet && other vxlan,vlan interface.
(the vnet is a bridge with same name)

That's why I have added an alias, where you can use a longer name, more descriptive. (I don't displaying yet in vm/ct nic bridge selection, I'll coming soon)

So, I don't known how to generate an unique id here.
Maybe I could add an option, to define an user custom pattern ? (like : vnet(\d+)), and auto increment counter for example.
I'm thinking to add some kind of batch vnet add wizared, where you can said add 1000 vnets with id from (1-1000).
But I can't used directly vni for the name, because vni can be really long (16millions).

 

[spirit]

I've tried this between two test nodes (not part of a cluster) and VXLAN, the traffic seems to pass through, but ping tests reports duplicated answers:

View attachment 20310

Any hints?

mmm, this is strange, I never see this.
Can you send your /etc/pve/sdn/*.cfg && /etc/network/interfaces && /etc/network/interfaces.d/sdn of both nodes ?

 

[cyruspy]

Hi, I can't use uuid where, because of limitation of linux interfaces name lenght. (14caracters), and I'm currently limiting vnet to 8 caracters, because I'm also use some internal mapping between vnet && other vxlan,vlan interface.
(the vnet is a bridge with same name)

That's why I have added an alias, where you can use a longer name, more descriptive. (I don't displaying yet in vm/ct nic bridge selection, I'll coming soon)

So, I don't known how to generate an unique id here.
Maybe I could add an option, to define an user custom pattern ? (like : vnet(\d+)), and auto increment counter for example.
I'm thinking to add some kind of batch vnet add wizared, where you can said add 1000 vnets with id from (1-1000).
But I can't used directly vni for the name, because vni can be really long (16millions).

Well, a pseudo UUID would work also . my suggestion would be just to autogenerate the ID, I don't see a use case to define the pattern by the user.

I'm just thinking about how to make it less possible to clash ("XX VNI is already used" or "VNET ID already used"). ID shouldn't be used as descriptive (asked to the user), that's what you have Alias for.

VNI generation would be nice, but I do see a use case where the user would need to match it with what he/she has on the other side (external to your control?)

 

[cyruspy]

mmm, this is strange, I never see this.
Can you send your /etc/pve/sdn/*.cfg && /etc/network/interfaces && /etc/network/interfaces.d/sdn of both nodes ?

Attaching files...

 

[spirit]

Well, a pseudo UUID would work also . my suggestion would be just to autogenerate the ID, I don't see a use case to define the pattern by the user.

I'm just thinking about how to make it less possible to clash ("XX VNI is already used" or "VNET ID already used"). ID shouldn't be used as descriptive (asked to the user), that's what you have Alias for.

Well, I could generate a simple "vnet(\d+)" default value in the form for the vnetid. (with an autoincrement, so I easily find available vnet).

VNI generation would be nice, but I do see a use case where the user would need to match it with what he/she has on the other side (external to your control?)

I think I could just pre-fill the form too with next vni available. (User can still change it), but for vlan for example, user need to be able to define it to match is physical vlans.
I think it's be more confusing.

I need to think about this more deeply, thanks for your comments !

 

[spirit]

Attaching files...

sdn config seem fine,

I have question about this on bigiron node

auto vmbr1
iface vmbr1 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1
    bridge-ports enp6s5f0 enp6s5f1 vmbr0.100
    bridge-stp off
    bridge-fd 0

Why do you have vmbr0.100 in vmbr1 ?

also why do you have 2 physical interfaces (enp6s5f0, enp6s5f1) in this bridge ?
(I think are plugged on same physical switch, you'll have a loop. you should create a bond interface in this case)

 

[cyruspy]

sdn config seem fine,

I have question about this on bigiron node

auto vmbr1
iface vmbr1 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1
    bridge-ports enp6s5f0 enp6s5f1 vmbr0.100
    bridge-stp off
    bridge-fd 0

Why do you have vmbr0.100 in vmbr1 ?

also why do you have 2 physical interfaces (enp6s5f0, enp6s5f1) in this bridge ?
(I think are plugged on same physical switch, you'll have a loop. you should create a bond interface in this case)

Hello!,

bigiron is in little distant from the switch. I use additional ports in that node to connect other clients as a cascaded switch.

Those physical ports are connected to a PDU and a UPS for monitoring purposes. Those ports are in access, while they need to participate in the management VLAN (VLAN 100 in the port connected to the switch)

That is why

 

[spirit]

Hello!,

bigiron is in little distant from the switch. I use additional ports in that node to connect other clients as a cascaded switch.

Those physical ports are connected to a PDU and a UPS for monitoring purposes. Those ports are in access, while they need to participate in the management VLAN (VLAN 100 in the port connected to the switch)

That is why

mmm, ok, I think I understand.
Why not simply do , instead your conf:

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet manual
    vlan-id 100


auto vmbr1
iface vmbr1 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1
    bridge-ports enp6s5f0 enp6s5f1 vmbr0.100
    bridge-stp off
    bridge-fd 0
#Downlink para gest

this (tagging vlan on physical interfaces)

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0  enp6s5f0.100 enp6s5f1.100
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1

or using vlan-aware feature (with ifupdown2 package)

auto enp6s5f0
iface enp6s5f0
    bridge-access 100

auto enp6s5f1
iface enp6s5f1
    bridge-access 100

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s4f0  enp6s5f0 enp6s5f1
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094

auto vmbr0.100
iface vmbr0.100 inet static
    address 10.2.0.223/24
    gateway 10.2.0.1

 

[alyarb]

Is anyone using this in production, even for the simple VLAN use case?

I can't seem to create a zone and net that doesn't get the warning icon. The pve gui bugs out forcing a full screen refresh, and VMs will not start that are attached to my net:


bridge 'testNet' does not exist
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
TASK ERROR: start failed: QEMU exited with code 1

 

[spirit]

Is anyone using this in production, even for the simple VLAN use case?

I can't seem to create a zone and net that doesn't get the warning icon. The pve gui bugs out forcing a full screen refresh, and VMs will not start that are attached to my net:


bridge 'testNet' does not exist
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
TASK ERROR: start failed: QEMU exited with code 1

Hi,
that's sound like the config is not correctly apply, and bridge/vnet "testNet" is not created.
are you sure to have used the apply button on the main sdn section part ?

if yes, can you provide /etc/pve/sdn/*.cfg , /etc/nework/interfaces , /etc/network/interfaces.d/sdn config files ?

 

[alyarb]

yikes, I don't know how I could have missed that. sorry. thanks for the great module.

when people outgrow the simple VLAN and have to go for more encapsulation, how do they handle reducing the mtu on thousands of nics? is there any real performance hit by lowering the mtu for the other modes?

 

[jlebherz]

Hey,

we are running our cluster with Jumbo-Frames (MTU 9000) so we can run every VXLAN with the standard MTU of 1500 inside the VXLAN

 

[spirit]

yikes, I don't know how I could have missed that. sorry. thanks for the great module.

when people outgrow the simple VLAN and have to go for more encapsulation, how do they handle reducing the mtu on thousands of nics? is there any real performance hit by lowering the mtu for the other modes?

I'm personnaly using bigger mtu on physical interfaces (I can because It's my own network).

If you can't , 2 possibity: reduce mtu inside the guests os , or on qemu, it's possible to add ",mtu=...." to nic option (virtio nic only). Not yet available in gui yet.

I'm currently looking to auto add this option when mtu is defined on vnet, should be available soon.

 

[alyarb]

thanks guys. I will go with jumbo frames on the physical network as well.

 

[alyarb]

VLAN mode is working well in terms of creating/applying the config.

So I create a VXLAN zone called "SDN" with MTU 8950 and all my hosts' vmbr0 addresses in the peer list.

Then I create a net called testNet, tag 9999 and the rest on auto. When I hit apply, the "pending" zone turns to error.

Where can I see the error message?

Hotplug changing the NIC's bridge does not work. Tested on Windows and pfSense VM. Qemu-ga is working, but should not be required.

Hotplug error:

Parameter verification failed. (400)

net3: hotplug problem - can't add ovs port 'tap9902i3' - command '/usr/bin/ovs-vsctl -- add-port testNet tap9902i3' failed: open3: exec of /usr/bin/ovs-vsctl -- add-port testNet tap9902i3 failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 444.