[SOLVED] CT/VM (+br/vlan) issues post 6 to 7 upgrade

jaxx

Member
Oct 11, 2017
16
0
21
42
Burgundy, France
Hi all,

I'm having a bit of a pickle in the way of fully upgrading my cluster to pve 7.

Beyond having to fiddle my interface options (underscores being replaced by hyphens on bond interfaces, didn't expect that one ) and managing to get ifupdown2 installed _after_ being shut off from the world (thank god IPKVM exists, though ours still use java ):

Hypervisors work just fine by themselves, ssh/gui up via GigE bonds/bridges, ceph syncs fine and so does the clustering stack over a bonded double 10G loop between the three servers (we have our own racks, but barely starting to put 10G equipement in), PVE 6 and 7 don't have any issue talking to each other.

10G cards are dual-port intel X520 (ixgbe)

But, I'm having a hard time grasping why vlan'd CT's and VM's have zero chatter to and from the PVE 7 host (though public IP seems ok, it wasn't working initally for some reason).

I can see ARP requests from CT's on the PVE 6 hosts arriving on bridge10.410 (using tcpdump), but not on vmbr410 (which only bridges bridge10.410) ‍

I did have hwaddress set in the configuration but having ipupdown2 I took them out (with no effect)

Also tried turning tso/gso/etc... off with ethtool just in case

The stack is:

CT with vlan tagged interface on vmbr410
vmbr410 bridge_vlan_aware yes
bridge10.410
bridge10
bond100bond101
bond-slaves enp130s0f0 enp132s0f0bond-slaves enp130s0f1 enp132s0f1

Here's the relevant configuration:

Code:
auto enp130s0f0
iface enp130s0f0 inet manual
  mtu 9000

auto enp132s0f0
iface enp132s0f0 inet manual
  mtu 9000

auto bond100
iface bond100 inet manual
  bond-slaves enp130s0f0 enp132s0f0
  bond-min-links 1
  mtu 9000
  bond-miimon 100
  bond-mode 802.3ad

auto enp130s0f1
iface enp130s0f1 inet manual
  mtu 9000

auto enp132s0f1
iface enp132s0f1 inet manual
  mtu 9000

auto bond101
iface bond101 inet manual
  bond-slaves enp130s0f1 enp132s0f1
  bond-min-links 1
  mtu 9000
  bond-miimon 100
  bond-mode 802.3ad

auto bridge10
iface bridge10 inet manual
  bridge-ports bond100 bond101
  bridge-stp on
  mtu 9000

auto bridge10.410
iface bridge10.410 inet manual
    mtu 9000

auto vmbr410
iface vmbr410 inet manual
    bridge_ports bridge10.410
    bridge_stp off
    bridge_fd 0
    bridge_vlan_aware yes
    mtu 9000


Vlan 410 is for inter CT/VM communication, each 'client' gets his own VLAN tag on top of the bridge. 498/499 are used for the clustering ring and storage ring (ceph here)
Vlan 601, is a simpler vmbr601 on bond0.601, switchs in front, for public IP space.
vmbr0 (on bond0) remains in the native vlan for administration

Notification_Center-7.png

CTs and VMs work perfectly fine on the PVE6 hosts

(I know mtu 9000 looks odd knowing there are vlan stacks to count, but at least if I got a simple ping through, I'd be happy right now)

Code:
root@xxxxx-priv-01:~# pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-3-pve)
pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e)
pve-kernel-5.11: 7.0-6
pve-kernel-helper: 7.0-6
pve-kernel-5.4: 6.4-5
pve-kernel-5.3: 6.1-6
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph: 15.2.14-pve1
ceph-fuse: 15.2.14-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.3.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-6
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.9-2
proxmox-backup-file-restore: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.0-9
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-13
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
root@xxxxx-priv-01:~#

I have also encountered a slight issue when hotplugging interfaces to a CT (haven't tested VMs) where the GUI throws an error but sort of writes the configuration anyhow:
Notification_Center-4.png
Code:
root@xxxxx-priv-01:~# /sbin/ip link add name veth103357010i2 mtu 9000 type veth peer name veth103357010i2p mtu 9000 addr 6A:24:80:2B:C7:C2
Error: argument "veth103357010i2p" is wrong: "name" not a valid ifname
root@xxxxx-priv-01:~# dpkg -S /sbin/ip
iproute2: /sbin/ip
root@xxxxx-priv-01:~# apt list iproute2
Listing... Done
iproute2/stable,now 5.10.0-4 amd64 [installed]
root@xxxxx-priv-01:~#


The question isn't "if" I am doing something wrong... more of "What am I doing wrong" ? the lack of output on a simple tcpdump -nvvi vmbr410 make me wonder if the bridge is restricting vlans, do the ids need to be explicitly declared ? (where as they've never been before)

Thanks in advance for your input ;-)

JaXX./.
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,874
704
133
www.odiso.com
Code:
auto bridge10
iface bridge10 inet manual
  bridge-ports bond100 bond101
  bridge-stp on
  mtu 9000

auto bridge10.410
iface bridge10.410 inet manual
    mtu 9000

auto vmbr410
iface vmbr410 inet manual
    bridge_ports bridge10.410
    bridge_stp off
    bridge_fd 0
    bridge_vlan_aware yes
    mtu 9000

that's seem wrong, bridge-vlan-aware should be set on bridge10, to be able to create the bridge10.410 interface.

it should be like this:


Code:
auto bridge10
iface bridge10 inet manual
  bridge-ports bond100 bond101
  bridge-stp on
  bridge_vlan_aware yes
  mtu 9000

auto bridge10.410
iface bridge10.410 inet manual
    mtu 9000

auto vmbr410
iface vmbr410 inet manual
    bridge_ports bridge10.410
    bridge_stp off
    bridge_fd 0
    mtu 9000
    bridge_vlan_aware yes
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,874
704
133
www.odiso.com
Vlan 601, is a simpler vmbr601 on bond0.601, switchs in front, for public IP space.

I don't see it in your conf, but personnaly, I would avoid mixing vlan tag on vmbr vlan-aware && physical interfaces, I have already see differents bug because of this.
 

jaxx

Member
Oct 11, 2017
16
0
21
42
Burgundy, France
max interface name is 15 characters,
is your vm is "103357010" ? because it's too big.

Oh, cr*p ... I have >60 VM/CTs on the cluster (and a couple of /23 and a handful of /24s) for half a dozen clients, that's why my numbering scheme is vast ... it hopefully hinders only hotplugging, because I don't feel like renumbering everything :)

...

that's seem wrong, bridge-vlan-aware should be set on bridge10, to be able to create the bridge10.410 interface.

it should be like this:


Code:
auto bridge10
iface bridge10 inet manual
  bridge-ports bond100 bond101
  bridge-stp on
  bridge_vlan_aware yes
  mtu 9000

auto bridge10.410
iface bridge10.410 inet manual
    mtu 9000

auto vmbr410
iface vmbr410 inet manual
    bridge_ports bridge10.410
    bridge_stp off
    bridge_fd 0
    mtu 9000
    bridge_vlan_aware yes

Errrm... bridge10.410 is created fine, so are the .498 and .499 for clustering/storage and are in active use (via the vmbr498/vmbr499)... since I add a vlan tag per VM/CT on the top vmbr410 (who needs to be aware)

The cluster/storage bridges are as:
Code:
auto bridge10.498
iface bridge10.498 inet manual
    mtu 9000

auto vmbr498
iface vmbr498 inet static
    address 10.33.94.1
    netmask 255.255.255.0
    bridge_ports bridge10.498
    bridge_stp off
    bridge_fd 0

auto bridge10.499
iface bridge10.499 inet manual
    mtu 9000

auto vmbr499
iface vmbr499 inet static
    address 10.33.194.1
    netmask 255.255.255.0
    bridge_ports bridge10.499
    bridge_stp off
    bridge_fd 0

bridge10 is composed of two underlying bonds, they shouldn't appear to proxmox for VM/CT networks, and are solely used as a support to the three VMBRs 410/498/499 each in a separate vlan on bridge10

Hope it makes more sense.

I don't see it in your conf, but personnaly, I would avoid mixing vlan tag on vmbr vlan-aware && physical interfaces, I have already see differents bug because of this.

I didn't show this part because it's not giving any issue

Code:
auto vmbr0
iface vmbr0 inet static
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
    mtu 9000
    address 172.xx.yy.131
    netmask 255.255.255.0
    gateway 172.xx.yy.1
    dns-nameserver 172.xx.yy.53
    dns-nameserver 172.xx.yy.153
    dns-search pve.xxxxx.com

auto vmbr601
iface vmbr601 inet manual
    bridge_ports bond0.601
    bridge_stp off
    bridge_fd 0
    mtu 9000
 

jaxx

Member
Oct 11, 2017
16
0
21
42
Burgundy, France
Side note:
It isn't PVE7 generating the issue (nor a too recent kernel) but ifupdown2 that isn't creating bridges exactly the same way, it wasn't installed (or needed) before ... in the upgrade path, I include now ifupdown2 to be installed, and the issue arises even on a PVE6 with ifupdown2.

I will eventually get to the bottom of this

JB./.
 

spirit

Famous Member
Apr 2, 2010
5,874
704
133
www.odiso.com
So, it's working if ifupdown1 ?

could be interesting to see the result of "bridge -c vlan" with both ifupdown1 && ifupdown2 to compare tagging inside the bridges.


I'm not sure, but it seem to missing allowed vlans "bridge-vids 2-4094" in vlan-aware-bridge, it's mandatory for ifupdown2. (if ifupdown1, we had a proxmox hack auto adding it).
 
Last edited:

jaxx

Member
Oct 11, 2017
16
0
21
42
Burgundy, France
So, it's working if ifupdown1 ?

could be interesting to see the result of "bridge -c vlan" with both ifupdown1 && ifupdown2 to compare tagging inside the bridges.


I'm not sure, but it seem to missing allowed vlans "bridge-vids 2-4094" in vlan-aware-bridge, it's mandatory for ifupdown2. (if ifupdown1, we had a proxmox hack auto adding it).
Yeah,

It was a perfectly working solution with ifupdown1…

Already beer-time here, but I’ll scroll my terminals tomorrow, bridge vlan dev bridge10 showed the three vlans as expected if I remember well, i don’t remember for vmbr410 which was holding the tagged CTs/VMs, I’ll give it a shot… I did try to add bridge-vlan-aware to bridge10 (though this one isn’t directly used by pve) it crippled the cluster quite fast as storage went haywire…

I’ll try the bridge-vlans tomorrow though, I thought it remained unnecessary by default.

JaXX./.
 

spirit

Famous Member
Apr 2, 2010
5,874
704
133
www.odiso.com
Well...

explicit bridge-vids on vmbr410 did the trick :facepalm:
Thanks !

I guess holidays weren't long enough

JB./.
ok thanks for testing ! I'll update the official proxmox doc example, because it was missing too. (when the config is done with the gui, it's already correctly added)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!