Setting up vmbrs for Vms frustration issue

[ieronymous]

Hello

I am trying since yesterday the simple thing of creating an extra vmbr bridge and attach it to a Vm but whatever I try doesnt seem to work.
A little back info first.
My motherboard has a dual nic and I have modified the default vmbr0 to be based not on the port enp1 but the bond0 who has set up as enp1 enp2 and balance-rr mode.
Nothing else there. The vmbr0 now is based on bond0 and has a static ip from which I can access Proxmox gui also and if I set this network device for every VM it has internet access. My problem now is .....
I have setup 2 more vmbrs, vmbr1 and vmbr2 upon a 4nic card installed on the motherboard. So the final configuration is like this

Network adapters configuration in Proxmox gui
vmbr0 based on bond0 based on ports (enp7s0 enp0s25) bond0 in mode balance-rr -> pending change to LACP (802.3ad)
vmbr1 based on bond1 based on ports (enp131s0f0 enp131s0f1) bond1 in mode balance-rr -> pending change to LACP (802.3ad)
vmbr2 based on bond2 based on ports (enp132s0f0 enp132s0f1) bond2 in mode balance-rr -> pending change to LACP (802.3ad)

vmbr0 is the only interface that has except a static ip a Gateway also (all vmbrs are vlan aware in case that creates some kind of problem)
If vmbr0 is being assigned to the VM everything is ok
If I change the hardware properties of the VM -> Network Device -Edit and change the vmbr0 to vmbr1 first notice is that the MAC address stays the same and
the OS shows the network card as unedifying...... so no net access even if in status of the adapter seems to have taken ip address/gateway/dns correct values (no ping also). I thought that because the VM has been created with the vmbr0 it would be more wise to add a network device instead of just editing and changing the vmbr device from vmbr0->vmbr1. So I did create a new network interface and set vmbr1 model:VirtIo and the MAC by default changed to a new one which is nice. I though that this was the problem and started the VM with no luck of course.
Tried afterwards to delete the static ip address from the vmbr1 option window and let only the option that it is based on bon1 (i couldnt assign an IP to the bond1 which makes sense since bond 1 afterwards been assigned to vmbr1)
No matter what I try I cant get the VM to have net access with other vmbr except the vmr0

Any help would be much appreciated because I am ready to cut this damn server in half with a catana!!!!

Thank you in advance

PS The extra network card I have installed checked for proper working condition. After all if I assign ip address to the vmbrs i can access proxmox gui from 3 different ips (dont need that I was just checking this way the fail over by unplugging cables)

 

[mira]

I'm not sure I understand it correctly.
You want to add a separate bridge on a different subnet for your VMs to communicate with each other?

 

[ieronymous]

I'm not sure I understand it correctly.
You want to add a separate bridge on a different subnet for your VMs to communicate with each other?

Until the separate bridge part you are correct but why different subnet? Same subnet and the VM need to has net access. After this part several other Vms will be created on the other vmbr bridge still same subnet and all have net access as well. The ability to communicate each other I thinnk by default it is on and mostly needed in my situation

Bottom line is I need a SQL Server VM to have net access by a dedicated dual port (that is why the bond in order to increase the lanes and not the total Gbps since nic teaming doesn t work this way of summing ports, On the contrary making the network connection wider)

 

[ieronymous]

Still trying to figure out why this is happening .......................
At some point by disabling inside the VM the network adapter and bringing it back again seemed to do the trick (even though it isnt supposed to work this way) but for a short period of time. After a while not only vmbr1 has that yellow triangle upon the adapter but vmbr0 also.
What I noticed in syslog when starting the VM based on vmbr0:

 pve pvedaemon[3682]: start VM 100: UPID:pve:00000E62:00325ADA:6095BFC2:qmstart:100:root@pam:
 pve pvedaemon[15764]: <root@pam> starting task UPID:pve:00000E62:00325ADA:6095BFC2:qmstart:100:root@pam:
 pve systemd[1]: Started 100.scope.
 pve systemd-udevd[3692]: Using default interface naming scheme 'v240'.
 pve systemd-udevd[3692]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[3692]: Could not generate persistent MAC address for tap100i0: No such file or directory
 pve kernel: device tap100i0 entered promiscuous mode
 pve systemd-udevd[3692]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[3692]: Could not generate persistent MAC address for fwbr100i0: No such file or directory
 pve systemd-udevd[3695]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[3695]: Using default interface naming scheme 'v240'.
 pve systemd-udevd[3695]: Could not generate persistent MAC address for fwpr100p0: No such file or directory
 pve systemd-udevd[3696]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[3696]: Using default interface naming scheme 'v240'.
 pve systemd-udevd[3696]: Could not generate persistent MAC address for fwln100i0: No such file or directory
 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
 pve kernel: device fwln100i0 entered promiscuous mode
 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
 pve kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
 pve kernel: vmbr0: port 2(fwpr100p0) entered disabled state
 pve kernel: device fwpr100p0 entered promiscuous mode
 pve kernel: device bond0 entered promiscuous mode
 pve kernel: device enp0s25 entered promiscuous mode
 pve kernel: device enp7s0 entered promiscuous mode
 pve kernel: vmbr0: port 2(fwpr100p0) entered blocking state
 pve kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
 pve kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
 pve pvedaemon[15764]: <root@pam> end task UPID:pve:00000E62:00325ADA:6095BFC2:qmstart:100:root@pam: OK
 pve kernel: fwbr100i0: received packet on fwln100i0 with own address as source address (addr:8a:fd:ad:5d:e3:ee, vlan:0)

Maybe some modification needs to be done at switch side also when selecting balance-rr ? Even so, isnt supposed to work at least one port if the other is in disable / stop state?

/etc/network/interfaces

auto lo
iface lo inet loopback

auto enp7s0
iface enp7s0 inet manual

auto enp0s25
iface enp0s25 inet manual

auto enp131s0f0
iface enp131s0f0 inet manual

auto enp131s0f1
iface enp131s0f1 inet manual

auto enp132s0f0
iface enp132s0f0 inet manual

auto enp132s0f1
iface enp132s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp0s25 enp7s0
        bond-miimon 100
        bond-mode balance-rr

auto bond1
iface bond1 inet manual
        bond-slaves enp131s0f0 enp131s0f1
        bond-miimon 100
        bond-mode balance-rr

auto bond2
iface bond2 inet manual
        bond-slaves enp132s0f0 enp132s0f1
        bond-miimon 100
        bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.201/24
        gateway 192.168.1.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr2
iface vmbr2 inet static
        address 192.168.1.203/24
        bridge-ports bond2
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

qm config 100

agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 4
ide0: local:iso/virtio-win-0.1.190.iso,media=cdrom,size=489986K
ide2: local:iso/winserv2019.iso,media=cdrom
machine: pc-i440fx-5.2
memory: 8192
name: WS2K19-DC
net0: virtio=A2:E4:72:4B:A9:7D,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: pveVM:vm-100-disk-0,cache=writeback,discard=on,size=300G
scsihw: virtio-scsi-pci
smbios1: uuid=deb91d48-f382-43b3-8f00-25fe55304136
sockets: 1
vmgenid: 65555f3e-1c08-45c8-bbc2-45b018f09c8b

PS I have already seen similar posts and answers / observations like @Alwin 's
<<These are the messages from the bridge on the STP state of the port connected.
They tell you if you have a loop somewhere in your network. And if STP works correctly, it will block a link to resolve the loop.>>
But that doesnt solve the problem. By the way what is STP state? When you say a loop in the system a more phrasal way to explain it?

 

[ieronymous]

I think I found something here and also something most with similar problems omit to include in their info. After changing the bond type from balance-rr to balance-alb, VM starting with internet access at last and following syslog

 pve pvedaemon[9624]: start VM 101: UPID:pve:00002598:0000A4A5:6096512A:qmstart:101:root@pam:
 pve systemd[1]: Created slice qemu.slice.
 pve systemd[1]: Started 101.scope.
 pve systemd-udevd[9631]: Using default interface naming scheme 'v240'.
 pve systemd-udevd[9631]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[9631]: Could not generate persistent MAC address for tap101i0: No such file or directory
 pve kernel: device tap101i0 entered promiscuous mode
 pve systemd-udevd[9631]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[9631]: Could not generate persistent MAC address for fwbr101i0: No such file or directory
 pve systemd-udevd[9631]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[9631]: Could not generate persistent MAC address for fwpr101p0: No such file or directory
 pve systemd-udevd[9633]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
 pve systemd-udevd[9633]: Using default interface naming scheme 'v240'.
 pve systemd-udevd[9633]: Could not generate persistent MAC address for fwln101i0: No such file or directory
 pve kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
 pve kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
 pve kernel: device fwln101i0 entered promiscuous mode
 pve kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
 pve kernel: fwbr101i0: port 1(fwln101i0) entered forwarding state
 pve kernel: vmbr2: port 2(fwpr101p0) entered blocking state
 pve kernel: vmbr2: port 2(fwpr101p0) entered disabled state
 pve kernel: device fwpr101p0 entered promiscuous mode
 pve kernel: device bond2 entered promiscuous mode
 pve kernel: device enp132s0f0 entered promiscuous mode
 pve kernel: vmbr2: port 2(fwpr101p0) entered blocking state
 pve kernel: vmbr2: port 2(fwpr101p0) entered forwarding state
 pve kernel: fwbr101i0: port 2(tap101i0) entered blocking state
 pve kernel: fwbr101i0: port 2(tap101i0) entered disabled state
 pve kernel: fwbr101i0: port 2(tap101i0) entered blocking state
 pve kernel: fwbr101i0: port 2(tap101i0) entered forwarding state
 pve kernel: kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable

So what I am wondering here is
-this port 2 entering blocking state first the forwarding afterwards is it something like a self test? (By the way I didnt have any other messages after
the initial ones)
-Can I do something about the <<created on host with unstable TSC; guest TSC will not be reliable>> message?
-Do all balance modes in bond type require a modification to the switch also? (At least I know they do with 802.3ad type, but what about the others?)
-Maybe when selecting Virtio Paravirtualized as nic adapter and with the current KVM 64 drivers some modes are problematic or something?

I can t think of anything else, out of ideas

 

[spirit]

-Do all balance modes in bond type require a modification to the switch also? (At least I know they do with 802.3ad type, but what about the others?)

The main problem with mode other than 802.3ad or active-backup, is that you can have for 1 tcp connection, packets balanced on 2 differents interfacces, and you can have out of order packets at the switch side, so retransmit.
almost any swich vendor support lacp, some vendor support other mode (cumulus linux support balance-xor), but generaly it need configuration at the switch side, if you really don't want 0 loss/retransmit.

https://www.kernel.org/doc/Documentation/networking/bonding.txt

5. Switch Configuration
=======================

    For this section, "switch" refers to whatever system the
bonded devices are directly connected to (i.e., where the other end of
the cable plugs into).  This may be an actual dedicated switch device,
or it may be another regular system (e.g., another computer running
Linux),

    The active-backup, balance-tlb and balance-alb modes do not
require any specific configuration of the switch.

    The 802.3ad mode requires that the switch have the appropriate
ports configured as an 802.3ad aggregation.  The precise method used
to configure this varies from switch to switch, but, for example, a
Cisco 3550 series switch requires that the appropriate ports first be
grouped together in a single etherchannel instance, then that
etherchannel is set to mode "lacp" to enable 802.3ad (instead of
standard EtherChannel).

    The balance-rr, balance-xor and broadcast modes generally
require that the switch have the appropriate ports grouped together.
The nomenclature for such a group differs between switches, it may be
called an "etherchannel" (as in the Cisco example, above), a "trunk
group" or some other similar variation.  For these modes, each switch
will also have its own configuration options for the switch's transmit
policy to the bond.  Typical choices include XOR of either the MAC or
IP addresses.  The transmit policy of the two peers does not need to
match.  For these three modes, the bonding mode really selects a
transmit policy for an EtherChannel group; all three will interoperate
with another EtherChannel group.

 

[ieronymous]

The active-backup, balance-tlb and balance-alb modes do not require any specific configuration of the switch

.... and that is why the problem with virtual nic adapter of the OS shows up connected directly after switching bond mode to balance-alb.

The balance-rr, balance-xor and broadcast modes generally require that the switch have the appropriate ports grouped together.

....and that is what I didnt doat first place to the switch side because I setup proxmox from a distance and up until tomorrow dont have physical access to the building, The final goal here is to make 3 dual (802.3ad type) connections and divide VMs upon these adapters accordingly.