[SOLVED] New VMs have no connectivity after upgrade - a tale of user error

T

thomps

New Member
Apr 10, 2020
8
0
1
38
Hello,

I recently upgraded to 6.1 and noticed that none of my new VMs can connect to any VLAN. There were no changes to the network environment. (this is my homelab - fyi)

Kernel: 5.3.18-3-pve
Proxmox: 6.1-8

Here are some things I've tried:
  • Booting an older kernel
  • Physically disconnecting one of the two bonded ports (the host system connects out to the network just fine irrespective of what's connected here)

I didn't change any configuration just yet. I wanted to hear some thoughts first before I ignorantly make a mess.

Thank you.

FILE: /etc/network/interfaces
Code: 
auto lo
iface lo inet loopback

# Physical Devices
# ================
iface enp2s0 inet manual
iface enp3s0 inet manual

# Network Bond
# ============
auto bond0
iface bond0 inet manual
        slaves enp2s0 enp3s0
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer2+3


# Guest Bridge Configuration
# ===========================
auto vmbr0
iface vmbr0 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes

# Host Configuration
# Note: We configure the host on VLAN 5.
auto vmbr0.5
iface vmbr0.5 inet static
        address 192.168.2.50
        netmask 255.255.255.0
        gateway 192.168.2.1

Possibly Related Posts:
 
Last edited:
Config seem to be ok. (maybe it's missing "bridge-vids 2-4094 to defined allowed vlans)

what is your nics models ?

for example, they are some problems with mellanox connectx-3 with last kernels, where they are limited to 128 vlans offloading. (and bridge-vids option need to have a reduce vlan ids range)
 
NICs: Intel 82574L

TEST:
I added the following to /etc/network/interfaces
bridge_vids 2-4094

OBSERVATIONS:
  • [NO CHANGE] New VMs have no connectivity
  • I noticed that I can create VMs which connect to VLAN 5. I expect this is because of the vmbr0.5.
 
I'm having the same issue over https://forum.proxmox.com/threads/vm-vlan-issue.68218/

I just did a upgrade on my 3 nodes and now they can't ping each other. This has never happened in the past and I confirmed that nothing in the config or switch has changed.

Reboot the 3 nodes and now they can ping each other but vlans are not working.
I would also like to add:
Code: 
bridge_vids 1-4094
instead of
Code: 
bridge_vids 2-4094

So VMs will need to access the native / untagged vlan on the vmbr0 bridge while others will be connecting to specific vlans using the tag option in the VM config.
 
Last edited:
cyrus104 said:
I'm having the same issue over https://forum.proxmox.com/threads/vm-vlan-issue.68218/

I just did a upgrade on my 3 nodes and now they can't ping each other. This has never happened in the past and I confirmed that nothing in the config or switch has changed.

Reboot the 3 nodes and now they can ping each other but vlans are not working.
I would also like to add:
Code: 
bridge_vids 1-4094
instead of
Code: 
bridge_vids 2-4094

So VMs will need to access the native / untagged vlan on the vmbr0 bridge while others will be connecting to specific vlans using the tag option in the VM config.
Click to expand...
native vlan 1 is already defined by default (bridge-vids don't define native vlans).
you can change the native vlan with "bridge-pvid X", but it's supported with ifupdown2 only.
 
TEST 1: Adjust vid range on bridge configuration
  • Edit /etc/network/interfaces
    • bridge_vids 2-4094 -> bridge_vids 2-10
  • Started a guest VM on VLAN 8

OBSERVATIONS:
  • Guest VM did not get assigned and IP by the local DHCP server
  • After manually assigning an IP to the same NIC, guest VM could not reach any neighbors on that same VLAN

TEST 2: Disable VLAN Offloading
  • Same conditions as test 1, but with vlan offloading disabled on host system
Code: 
[/INDENT]
[INDENT]ethtool -K enp2s0  rxvlan off txvlan off[/INDENT][/INDENT][/INDENT]
[INDENT]ethtool -K enp3s0 rxvlan off txvlan off[/INDENT][/INDENT][/INDENT]
[INDENT]

OBSERVATIONS:
  • [No change] I changed this setting to no affect
  • The setting appears to clear after reboot. As you would probably expect, no affect at that point either

I will try updating to the 5.4 kernel and follow up with you here.
 
Last edited:
Okay, i'm now on 5.4.27-1-pve. No change in guest vlan connectivity.

As a side note: For the past few tests I had been using a vm which was created on VLAN 8 specifically for testing these connectivity issues. I don't recreate the VM for each test. I'm not sure if that would have much of an impact on the test results. Just now I tried creating a fresh VM on VLAN 8 after having updated the kernel to 5.4. Still, no change in connectivity. I've even moved the VM between VLANs and still no results.
 
Damned, I'm out of ideas...

do you have tried without vlan aware ?

Code: 
# Guest Bridge Configuration
# ===========================
auto vmbr0
iface vmbr0 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0

# Host Configuration
# Note: We configure the host on VLAN 5.
auto vmbr0v5
iface vmbr0v5 inet static
        address 192.168.2.50
        netmask 255.255.255.0
        gateway 192.168.2.1
        bridge_ports bond0.5
        bridge_stp off
        bridge_fd 0


(without vlan-aware, proxmox will create vmbr0vX with bond0.X when you use vlan in a vm nic config)
 
I thank you for trying. I have not yet tested a network configuration without VLAN aware. I'll give it a go this evening.
 
I just tried this with VLAN AWARE disabled. No dice. I'm thinking I may try to build a fresh node and see if I get any different results there. If this were to work I still wonder which way would be the best practice:
  • Original configuration, or
  • this configuration with the vlan aware disabled?
 
thomps said:
I just tried this with VLAN AWARE disabled. No dice. I'm thinking I may try to build a fresh node and see if I get any different results there. If this were to work I still wonder which way would be the best practice:
  • Original configuration, or
  • this configuration with the vlan aware disabled?
Click to expand...

vlan aware is more clean, if you have a lot of vlans, you don't have a lot of vmbrXvY bridge generated.

But that pretty strange that it don't work for you in both case. Are you sure that your physical switch port is correctly configured ?
 
The switch port sends all vlans to the host machine. I do wonder about what may have happened on the network however. The last work which took place was a firmware upgrade and a firewall update. At this point it certainly sounds like something wonky/embarrassing may be going on.

Let's review. Vlan 5 is normal. May be that has something to do with the interface config or may be not.

On vlan 7 I get strange behavior as well. I have other guests with static ip assignments on vlan 7 that behave normally. I took an existing guest and moved it over there / changed the ip and that worked normally as well. I created a new guest and it didn't work. It's now looking like the new guest on vlan 7 was configured with a bad name server. (There is no DHCP on vlan 7) I'm building up an additional new guest there now to verify that idea... VERIFIED: VLAN 7 is fine.

I can't ping the gateway on vlan 8. That could be explained with a tagging issue on the switch. For vlan8 there is a DHCP server, but I can't reach it. I'm investigating that as I write. ... :eek: Vlan 8 is excluded. New guest auto configuration successful. Ugh. That's a paddling.

Well, this one is solved. #PEBKAC

I'll normalize the host interface config to the clean state.
 
@thomps, Can you post your host interface config when you get a chance. I'm currently not using vlan aware and looking to make the change over it from where I'm at. I have a hand full of VLANs that I have configured.

Thanks
 
@cyrus104, I'm now back to my original network configuration which includes the vlan aware bridge. In my current setup I have two network ports which are bonded and used in the guest bridge. If you're looking to use the physical port directly in your bridge you can probably bridge_ports line to point to some thing other than bond0.

Code: 
auto lo
iface lo inet loopback

# Physical Devices
# ================
iface enp2s0 inet manual
iface enp3s0 inet manual

# Network Bond
# ============
auto bond0
iface bond0 inet manual
        slaves enp2s0 enp3s0
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer2+3


# Guest Bridge Configuration
# ===========================
auto vmbr0
iface vmbr0 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes

# Host Configuration
# Note: We configure the host on VLAN 5.
auto vmbr0.5
iface vmbr0.5 inet static
        address 192.168.2.50
        netmask 255.255.255.0
        gateway 192.168.2.1
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom