Trunked ports network to VMs

[unsichtbarre]

Howdy Noob here,

I am sure this question comes up often, but I have searched and browsed and can't come up with a solution.

My network is all trunk ports, VLAN tag required, except for iSCSI which are access ports. Therefore I set up a bond (rr) for all of the trunk ports and then a Linux VLAN for the management IP of my PVE host. So far so good. Then I set up a bridge to the bond, VLAN aware, no IP or gateway.

Next I created a VM (Windows with VirtIO installed) and set the NIC to VLAN 26 (correct VLAN) and I get nothing. NIC shows as connected but no traffic passing.

What am I doing wrong?

THX in ADV,
-JB

Here's the config:

auto lo
iface lo inet loopback

auto ens161
iface ens161 inet manual

auto ens192
iface ens192 inet manual

auto ens193
iface ens193 inet manual

auto ens224
iface ens224 inet manual

auto ens225
iface ens225 inet static
        address 10.0.0.201/24
        mtu 9000
#iSCSI 1

auto ens256
iface ens256 inet static
        address 10.0.0.221/24
        mtu 9000
#iSCSI 2

auto bond0
iface bond0 inet manual
        bond-slaves ens161 ens192 ens193 ens224
        bond-miimon 100
        bond-mode balance-rr
#10GbE Bond for Trunk

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto MGMT
iface MGMT inet static
        address 172.20.100.201/16
        gateway 172.20.0.1
        vlan-id 26
        vlan-raw-device bond0
#Management IP

source /etc/network/interfaces.d/*

 

[cwt]

First shot in the dark: port/LAG config of your switch. What switch do you use? Balance RR or weighted balance rr is not a real network „standard“. Does your switch support LAGs/LACP?

 

[unsichtbarre]

Thanks @cwt ! In fairness, this is a nested PVE host on ESXi using a dVswitch with trunk ports set to allow: Promiscuous, MAC Changes, Forged Transmits. This network runs nested ESXi and nested VMware VMs (and VM network) without issue.

I have set the bond to active/backup and still no luck. With the bond on active/backup, the Linux VLAN for MGMT works on VLAN 26 (as it did on rr), which is why I am puzzled about the bridge and/or VMs not getting access to network.

THX in ADV,
-JB

Here is a VM config (VirtIO installed, but I have tried all the others too):


Here is my new network:

auto lo
iface lo inet loopback

auto ens161
iface ens161 inet manual

auto ens192
iface ens192 inet manual

auto ens193
iface ens193 inet manual

auto ens224
iface ens224 inet manual

auto ens225
iface ens225 inet static
        address 10.0.0.201/24
        mtu 9000
#iSCSI 1

auto ens256
iface ens256 inet static
        address 10.0.0.221/24
        mtu 9000
#iSCSI 2

auto bond0
iface bond0 inet manual
        bond-slaves ens161 ens192 ens193 ens224
        bond-miimon 100
        bond-mode active-backup
        bond-primary ens161
#10GbE Bond for Trunk

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10GbE Virtual Switch

auto MGMT
iface MGMT inet static
        address 172.20.100.201/16
        gateway 172.20.0.1
        vlan-id 26
        vlan-raw-device bond0
#Management IP
"/etc/network/interfaces" 64 lines, 1280 bytes

 

[unsichtbarre]

Further diagnosis, I disabled PVE firewall (pve-firewall stop) and immediately got a DHCP IP on Windows/VirtIO, but still not able to ping out except the PVE host (Linux VLAN) IP.

This may be related to the nested aspect of PVE, but why would VMware ESXi (nested) networking work and PVE no?

THX,
-JB

 

[unsichtbarre]

Ok, sorry to flood my own thread, but this is baffling:

From PVE CLI, able to ping: 172.20.10.101, 172.20.10.102, 172.20.10.103

From Windows PVE VM, able to ping: 172.20.10.101, 172.20.10.103 but not 172.20.10.102

172.20.10.101 and 172.20.10.103 are on the same ESXi host as PVE, 172.20.10.102 is on a different ESXi host (dVswitch, Promiscuous, MAC changes, forged transmits enabled).

So the PVE Linux VLAN (172.20.100.201/16) can ping all three IPs (on multiple ESXi hosts), however the Linux Bridge can only reach IPs on the same ESXi host as PVE. I have verified vSphere networking and cross-pinged 172.20.10.101, 172.20.10.102, 172.20.10.103 from each other, and there is no problem with vSphere network.

THX,
-John