Removing vlan id 1 from a trunk

[guerby]

Hi,

I'm trying to trunk a subset of the host VLANs to a VM using proxmox VE 6.3-3 on debian 10.

In the web UI there is only one option tag= for a VM interface which gives access to the given VLAN only as untagged inside the VM.
If no tag= is set then the VM sees all the VLAN configured on the bridge.

My host has 4 VLANs tagged 10 20 30 40.

I first edited manually /etc/network/interfaces since through the web UI you only get no VLAN or all VLAN (2-4095) unless I missed an option somewhere?

# cat /etc/network/interfaces
....
iface vmbr0 inet static
....
    bridge-vlan-aware yes
    bridge-vids 10 20 30 40

Then on the second interface of VM 101 I'm doing the following:

# qm set 101 --net1 virtio=96:50:38:74:17:27,bridge=vmbr0,trunks='10;20;30'
update VM 101: -net1 virtio=96:50:38:74:17:27,bridge=vmbr0,trunks=10;20;30
# bridge vlan
...
tap101i1     1 PVID Egress Untagged
     10
     20
     30
# bridge vlan del vid 1 dev tap101i1
# bridge vlan
...
tap101i1     10
     20
     30

By default VLAN 1 is passed untagged to the interface of the VM. I did remove it manually with the bridge vlan del command but could not find a way to do it within proxmox tools, am I missing something?

I tried to add a tag=none tag=0 tag=-1 but tag= accepts only integer from 1 to 4094 so it's useful if you want one of the VLAN of the trunk passed as untagged but not to remove vlan 1 untagged altogether.

Thanks for your help!

 

[guerby]

Looking at the source code of PVE it looks like adding untag 1 is done in all cases :

https://github.com/proxmox/pve-common/blob/master/src/PVE/Network.pm#L235

        $tag = 1 if !$tag;
        eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $tag, 'pvid', 'untagged']) };
        die "unable to add vlan $tag to interface $iface - $@\n" if $@;

 

[ph0x]

I guess VLAN 1 is - for Proxmox - just the PVID of your trunk regardless of the real PVID. Since that traffic is always untagged it does not matter much which number is assigned, does it?
Or would you want to remove untagged traffic at all? Not sure if this is possible.
Again, I recommend adding the VLANs as separate NICs, that spares you a lot of trouble. ;-)

 

[guerby]

I guess VLAN 1 is - for Proxmox - just the PVID of your trunk regardless of the real PVID. Since that traffic is always untagged it does not matter much which number is assigned, does it?
Or would you want to remove untagged traffic at all? Not sure if this is possible.
Again, I recommend adding the VLANs as separate NICs, that spares you a lot of trouble. ;-)

Yes I want to remove untagged traffic as it might be an unwanted communication channel between unrelated VM.

For example let's say a VM has trunk VLAN 10 20 and another one trunk VLAN 30 40 so in principle they're completely isolated at L2 ethernet. If proxmox keeps adding untagged VLAN 1 to both interfaces it creates an unwanted shared ethernet link between the two VM, in other word a security issue (security/isolation is one purpose of VLAN after all .

It is obviously possible via the "bridge vlan del vid 1 dev tap101i1" command in my first post and I think desirable to be able to control untagged traffic. Proxmox allow via tag= to control which VLAN is untagged (which is nice), but does not currently allow to remove VLAN 1 untagged when no untagged traffic is wanted.

I'm currently considering migrating from an hypervisor where trunk to VM are done properly without unwanted additions, if promox has no way to do it I would have to reconfigure all the VM in question which adds to the migration burden.

 

[ph0x]

How would you want to prevent untagged traffic?
Untagged traffic always lives inside the PVID and that is defined with the bridge.
You can't have vmbr0 have untagged traffic in some PVID for one machine and in another PVID for another machine.
If you want to separate the networks you have to define separate bridges.
And again, if you add your VLANs as separate NICs you just don't have this problem, since the VLAN traffic will be untagged then and Proxmox will add the tag in the PVID that is defined within your switch.

 

[guerby]

How would you want to prevent untagged traffic?
Untagged traffic always lives inside the PVID and that is defined with the bridge.
You can't have vmbr0 have untagged traffic in some PVID for one machine and in another PVID for another machine.
If you want to separate the networks you have to define separate bridges.
And again, if you add your VLANs as separate NICs you just don't have this problem, since the VLAN traffic will be untagged then and Proxmox will add the tag in the PVID that is defined within your switch.

Hi,

Yes of course you can have untagged traffic VLAN 10 on one machine and untagged traffic VLAN 20 on another, that's what the tag= attribute is for, and you do not have to use separate bridges for that since "VLAN aware bridge" have been developped just for this purpose.

Please reread the commands in my first post, they show a bridge port without untagged traffic after deleting vlan 1.

I aslo explained why I would like to avoid creating new interfaces in existing VM : because other hypervisors on the market do not have issue with doing a proper trunk port to VM, I would have to modify heavily the network configuration inside these VM when migrating to proxmox.

 

[ph0x]

I slowly understand where you want to go and I still think you're trying to achieve it in the wrong place. Maybe add different tags then instead of removing them completely? Maybe look at QinQ?

 

[guerby]

I slowly understand where you want to go and I still think you're trying to achieve it in the wrong place. Maybe add different tags then instead of removing them completely? Maybe look at QinQ?

QinQ is another story We don't use it in our infrastructure currently.

The tag= attribute can be given only once and it's ok since it specify the untagged vlan and there can be only one on a bridge port

The issue is that proxmox cannot remove the current behaviour of adding untagged vlan 1 inconditionnally when there's only trunk= (which would be natural) because it will potentially break proxmox user existing setup.

I will test a patch which 'hacks" tag=0 to not add vlan 1 when there are trunk port defined, currently looking at the code impact.

 

[spirit]

I think that 1 solution, could be to not force pvid=1 if it's already defined in the trunk

something like this should work:

   if ($vlan_aware) {

        eval { run_command(['/sbin/bridge', 'vlan', 'del', 'dev', $iface, 'vid', '1-4094']) };
        die "failed to remove default vlan tags of $iface - $@\n" if $@;

        my $nopvid = undef;

        if ($trunks) {
            my @trunks_array = split /;/, $trunks;
            foreach my $trunk (@trunks_array) {
                my ($start, $end) = split /-/, $trunk;
                $nopvid = 1 if ($start eq 1 && !$end) || ($start <= 1 && $end >=1);

                eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $trunk]) };
                die "unable to add vlan $trunk to interface $iface - $@\n" if $@;
            }
        } elsif (!$tag) {
            eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', '2-4094']) };
            die "unable to add default vlan tags to interface $iface - $@\n" if $@;
        }

        $tag = 1 if !$tag && !$nopvid;
        eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $tag, 'pvid', 'untagged']) };
        die "unable to add vlan $tag to interface $iface - $@\n" if $@;
   }

 

[guerby]

I think that 1 solution, could be to not force pvid=1 if it's already defined in the trunk

something like this should work:

   if ($vlan_aware) {

        eval { run_command(['/sbin/bridge', 'vlan', 'del', 'dev', $iface, 'vid', '1-4094']) };
        die "failed to remove default vlan tags of $iface - $@\n" if $@;

        my $nopvid = undef;

        if ($trunks) {
            my @trunks_array = split /;/, $trunks;
            foreach my $trunk (@trunks_array) {
                my ($start, $end) = split /-/, $trunk;
                $nopvid = 1 if ($start eq 1 && !$end) || ($start <= 1 && $end >=1);

                eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $trunk]) };
                die "unable to add vlan $trunk to interface $iface - $@\n" if $@;
            }
        } elsif (!$tag) {
            eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', '2-4094']) };
            die "unable to add default vlan tags to interface $iface - $@\n" if $@;
        }

        $tag = 1 if !$tag && !$nopvid;
        eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $tag, 'pvid', 'untagged']) };
        die "unable to add vlan $tag to interface $iface - $@\n" if $@;
   }

I tested your suggestion manually:

# bridge -c vlan
...
tap101i1     10
     20
     30
# bridge vlan add vid 1 dev tap101i1
# bridge -c vlan
...
tap101i1     1
     10
     20
     30

So your code will add tagged vlan 1 to the trunk according to iproute2 bridge (didn't know this was possible ).

I checked with ping6 from the host and tcpdump on the VM and packets still reaching the VM with vlan tag 1:

host# ping6 fe80::9450:38ff:fe74:1727%vmbr0 -c 1

vm# tcpdump -eni ens19
19:21:30.622210 94:c6:91:ae:fd:3e > 33:33:ff:74:17:27, ethertype 802.1Q (0x8100), length 90: vlan 1, p 0, ethertype IPv6, fe80::96c6:91ff:feae:fd3e > ff02::1:ff74:1727: ICMP6, neighbor solicitation, who has fe80::9450:38ff:fe74:1727, length 32

So back to my original remark: proxmox cannot change the behaviour of current configurations without potentially breaking existing customer setup (if one happens to use untagged traffic in trunk setup it will currently work), so we need either to use tag=0 for this particular case (was invalid before so we're not breaking anything), or use another explicit new flag just for this (eg: trunkonly=true).

 

[spirit]

I tested your suggestion manually:

# bridge -c vlan
...
tap101i1     10
     20
     30
# bridge vlan add vid 1 dev tap101i1
# bridge -c vlan
...
tap101i1     1
     10
     20
     30

So your code will add tagged vlan 1 to the trunk according to iproute2 bridge (didn't know this was possible ).

I checked with ping6 from the host and tcpdump on the VM and packets still reaching the VM with vlan tag 1:

host# ping6 fe80::9450:38ff:fe74:1727%vmbr0 -c 1

vm# tcpdump -eni ens19
19:21:30.622210 94:c6:91:ae:fd:3e > 33:33:ff:74:17:27, ethertype 802.1Q (0x8100), length 90: vlan 1, p 0, ethertype IPv6, fe80::96c6:91ff:feae:fd3e > ff02::1:ff74:1727: ICMP6, neighbor solicitation, who has fe80::9450:38ff:fe74:1727, length 32

So back to my original remark: proxmox cannot change the behaviour of current configurations without potentially breaking existing customer setup (if one happens to use untagged traffic in trunk setup it will currently work), so we need either to use tag=0 for this particular case (was invalid before so we're not breaking anything), or use another explicit new flag just for this (eg: trunkonly=true).

note that we also have the default pvid=1 on the vmbr0 bridge.

if you use ifupdown2 for network management, they are some other bridge option for vmbr0:

auto vmbr0
   bridge-allow-untagged no     #don't set any pvid
   bridge-pvid 1   # the default default
   bridge-vids 2-2094   # the allow vlan in the thrunk

(bridge-allow-untagged sound like tag=0 behaviour with proxmox code)

 

[guerby]

note that we also have the default pvid=1 on the vmbr0 bridge.

Yes but if we define proper trunks for VM ports it doesn't get through.

Since proxmox UI doesn't allow for specifying ifupdown2 advanced VLAN options and you have to edit manually /etc/network/interfaces to for example select some VLANs instead of all it's not proxmox responsability to use the wanted options there.

if you use ifupdown2 for network management, they are some other bridge option for vmbr0:

auto vmbr0
   bridge-allow-untagged no     #don't set any pvid
   bridge-pvid 1   # the default default
   bridge-vids 2-2094   # the allow vlan in the thrunk

(bridge-allow-untagged sound like tag=0 behaviour with proxmox code)

Yes I'd like an equivalent to ifupdown2 "bridge-allow-untagged no" in proxmox for VM ports (internally in bridge.py ifupdown2 use pvid=0 to mark unwanted untagged 1).

https://github.com/CumulusNetworks/ifupdown2/blob/master/ifupdown2/addons/bridge.py#L1577

pvid_int = int(pvid) if pvid else 0

 

[guerby]

I lightly tested the following patch and it seemed to work for my trunk VM port and not break my other VM:

root@nuc3:/usr/share/perl5/PVE# diff -u Network.pm.orig Network.pm; diff -u QemuServer.pm.orig QemuServer.pm
--- Network.pm.orig    2021-02-02 20:19:17.454498452 +0100
+++ Network.pm    2021-02-02 20:43:19.216949189 +0100
@@ -231,6 +231,8 @@
             die "unable to add default vlan tags to interface $iface - $@\n" if $@;
         }
 
+    return if defined($tag) && $tag == 0;
+    
         $tag = 1 if !$tag;
         eval { run_command(['/sbin/bridge', 'vlan', 'add', 'dev', $iface, 'vid', $tag, 'pvid', 'untagged']) };
         die "unable to add vlan $tag to interface $iface - $@\n" if $@;
--- QemuServer.pm.orig    2021-02-02 20:19:30.806856976 +0100
+++ QemuServer.pm    2021-02-02 20:18:40.921521050 +0100
@@ -882,7 +882,7 @@
     },
     tag => {
     type => 'integer',
-    minimum => 1, maximum => 4094,
+    minimum => 0, maximum => 4094,
     description => 'VLAN tag to apply to packets on this interface.',
     optional => 1,
     },

 

[albert_a]

Hi, you can specify different VLAN tags to different VM groups that you want to isolate as well as 'trunks='. Though I've never came across such use case myself, but for cutting untagged traffic completely your patch is a nice solution.