SDN - vxlan, vNets: VLAN Aware vs Subnets

[pille99]

hello all

i am wondering what is the advantages/disadvantages of following settings:

in the SDN i created vxlan. on the vNet configuration are 2 differnt options to configure
1. vNet without subnet but with vLAN Awarnet, added the NIC to the VM and add here a VLAN ID
2. vNet with subnets and no vLAN, on the VM i add the Subnet

i was wondering which option in respect of performance is a better choice, or any other advantages/disadvantages - except that with subnet is more clean configuration (from my point of view)

the vxLan will be connected to an opnsense - in case this information matters.

 

[pille99]

at least one point i know now
- with subnet i got again the MTU Issue (dear proxmox team, plz automate to find the best possible MTU value, and if necessary like vxlan, reduce the 50 byte automatically - it is not so difficult to program). why on every change (on the pysically or SDN) i face the MTU Issue ?
- with vLan it worked after 30 mins playing with the MTU. the MTU which worked: physically Adapter: 9000, zone: 8950
for any strange reason: i put on the Virtual NIC for the VM the MTU = 1, sometimes it worked, sometimes not, but the MTU value should be inherite (so, on the virtual NIC it should be 8950).

how it shows: the Internet explorer or any other Browser cant open properly a Service-WebPage on another Server.

 

[spirit]

hello all

i am wondering what is the advantages/disadvantages of following settings:

in the SDN i created vxlan. on the vNet configuration are 2 differnt options to configure
1. vNet without subnet but with vLAN Awarnet, added the NIC to the VM and add here a VLAN ID
2. vNet with subnets and no vLAN, on the VM i add the Subnet

i was wondering which option in respect of performance is a better choice, or any other advantages/disadvantages - except that with subnet is more clean configuration (from my point of view)

the vxLan will be connected to an opnsense - in case this information matters.

the vnet are mainly thinked without beeing vlan-aware . (vlan-aware is to defined an extra vlan tag in the vm nic gui, over the tag of the vnet). so double tag.
Some users wanted it (like pushing multiple vlan tag inside a vxlan vnet), that's why it's implemented.


Subnet can't be use with vlan-aware, because you can only have 1 subnet by vnet.
currently subnet are mainly used for evpn && routed setup, but in the future, ipam will be able to use it to distribute ip to the vms. (dhcp or static).

 

[spirit]

at least one point i know now
- with subnet i got again the MTU Issue (dear proxmox team, plz automate to find the best possible MTU value, and if necessary like vxlan, reduce the 50 byte automatically - it is not so difficult to program). why on every change (on the pysically or SDN) i face the MTU Issue ?

They are no relation between subnet and mtu, that's strange.

We don't want to reduce it from 50 bytes by default, because users can handle with mtu bigger than 1500. (on their own network, then can manage jumbo frame with 9200 for example). It's really not easy to implement an autofind best value, because you need to test each node vs all other nodes.


if you define 1450 on the zone, it should be pushed on all vnets.
and if you defined on the vm, a virtio net nic, with mtu=1 (or auto), it should inherited from the mtu of the vnet

They are no reason that it could break on sdn reload or config change if you don't change the vlan mtu config.

- with vLan it worked after 30 mins playing with the MTU. the MTU which worked: physically Adapter: 9000, zone: 8950
for any strange reason: i put on the Virtual NIC for the VM the MTU = 1, sometimes it worked, sometimes not, but the MTU value should be inherite (so, on the virtual NIC it should be 8950).

how it shows: the Internet explorer or any other Browser cant open properly a Service-WebPage on another Server.

mtu=1 use a new virtio-net feature to be able to setup mtu in the guest os, so it work with recent virtio-net driver.

What is your guest os ? (I'm seeing "internet explorer", so old windows guest ?)

 

[pille99]

They are no relation between subnet and mtu, that's strange.

We don't want to reduce it from 50 bytes by default, because users can handle with mtu bigger than 1500. (on their own network, then can manage jumbo frame with 9200 for example). It's really not easy to implement an autofind best value, because you need to test each node vs all other nodes.


if you define 1450 on the zone, it should be pushed on all vnets.
and if you defined on the vm, a virtio net nic, with mtu=1 (or auto), it should inherited from the mtu of the vnet

They are no reason that it could break on sdn reload or config change if you don't change the vlan mtu config.


mtu=1 use a new virtio-net feature to be able to setup mtu in the guest os, so it work with recent virtio-net driver.

What is your guest os ? (I'm seeing "internet explorer", so old windows guest ?)

did you understand which "subnet" i meant: SDN, vNets - Subnet

an autoscaller for MTU would make the life much more easier. i dont understand why not more Post for MTU are open. i guess the most people just leave it default (which is a performance degrees of 30%). what i did: i changed from physically NIC (direct connect) to Pysically NIC and in top an SDN, with zone and vNet. for any reason on the subnet i couldnt get it fly, so i tried the vLAN option and it worked with the settings mentioned in top

i had the issue with MTU the 4th time already. the first time it needed to be changed from 9000 to 8972 and it worked again - nobody knew why. usually a network infrastructure is setup one time and never touch again, or very rarely.

 

[spirit]

did you understand which "subnet" i meant: SDN, vNets - Subnet

yes, this one, they are no other subnet option.

Currently, the only thing that subnet is doing, is defined gateway ip defined on the subnet, on the vnet (for evpn anycast gateway, or with simple zone, for routed setup). That's all.

an autoscaller for MTU would make the life much more easier. i dont understand why not more Post for MTU are open. i guess the most people just leave it default (which is a performance degrees of 30%).

Personnaly, I don't think it's difficult. You just need 50 less bytes mtu than your physical network support, that's all.
It's clearly documented in sdn doc.

it's really not easy to automate test, because that's mean we need to test all hosts to all hosts, and if 1switch port or 1 switch is missconfigured, we could get wrong results.

what i did: i changed from physically NIC (direct connect) to Pysically NIC and in top an SDN, with zone and vNet. for any reason on the subnet i couldnt get it fly, so i tried the vLAN option and it worked with the settings mentioned in top

i really wonder if it's not a physical nic bug. (what is your nic model).

I running vxlan in production with mtu=9000 (and 9200 on my switches), and I never had problem like you. (mellanox connect-x4/5 cards)

i had the issue with MTU the 4th time already. the first time it needed to be changed from 9000 to 8972 and it worked again - nobody knew why. usually a network infrastructure is setup one time and never touch again, or very rarely.

not sure here, maybe it's related to your nic driver ?

note that if you test with a ping, ping use 28 bytes.

ping -s 1472 -Mdo if you have mtu=1500 without vxlan
ping -s 1422 -Mdo if you have mtu=1500 + vxlan

 

[pille99]

the NICs are nothing special

 

[spirit]

can you send your /etc/network/interfaces && /etc/pve/sdn/*.cfg ?