So I just got and started to setup a 40g network for my 3 node cluster.
I'm using connectx3 pro cards with dual qsfp ports.
here is my interfaces file:
The LXC container details:
From the rest of my network I can ping 192.168.240.30(my lan), 192.168.242.30, and 10.40.0.30(from a VM on another node on vlan4000)
This leads me to think my bridge is functioning correctly and vlan aware however I cannot ping 192.168.41.90 from anything except the hypervisor.
So far only one of the nodes has the connectx card, the rest of my nodes are still running 10g intel x710 cards until I get my 40g switch setup(currently I have a mikrotik switch which has 2 40g ports and the rest of the hosts are connected via its 10g ports)
brctl details
if I put the LXC on my other bridge which is the 2.5g intel nic on the computer everything works.
Maybe I'm not searching for the right thing but I can't find any other topics that seem to address this issue, as visible I have tried to disable vlan offloading without effect as well and from my understanding should not be needed since I'm specifying all the vlan's I want and they are well under the card limit(this was already required by the intel x710)
If I can provide anything else to help I'd be happy to.
Thank you in advance.
I'm using connectx3 pro cards with dual qsfp ports.
here is my interfaces file:
Code:
auto lo
iface lo inet loopback
auto enp6s0
iface enp6s0 inet manual
mtu 9000
auto enp1s0
iface enp1s0 inet manual
mtu 9000
post-up ip link set dev enp1s0 promisc on
post-up /bin/bash -c 'for i in {0..3}; do /usr/sbin/ip link set dev enp1s0 vf $i spoof off promisc on; done'
auto enp1s0d1
iface enp1s0d1 inet manual
mtu 9000
post-up ip link set dev enp1s0d1 promisc on
post-up /bin/bash -c 'for i in {0..3}; do /usr/sbin/ip link set dev enp1s0d1 vf $i spoof off promisc on; done'
auto enp1s0v0
iface enp1s0v0 inet manual
mtu 9000
post-up ip link set dev enp1s0v0 promisc on
auto enp1s0d1v0
iface enp1s0d1v0 inet manual
mtu 9000
post-up ip link set dev enp1s0d1v0 promisc on
iface enp1s0d1v1 inet manual
iface enp1s0d1v2 inet manual
iface enp1s0d1v3 inet manual
iface enp1s0f3 inet manual
iface enp1s0v1 inet manual
iface enp1s0v3 inet manual
iface enp1s0v2 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp1s0d1v0 enp1s0v0
bond-miimon 100
bond-mode balance-alb
mtu 9000
post-up ip link set dev bond0 promisc on
post-up /sbin/ethtool -K bond0 rxvlan off
post-up /sbin/ethtool -K bond0 rx-vlan-offload off
auto vmbr1
iface vmbr1 inet manual
bridge-ports enp6s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2,3,10,30,40,41,71,90,100,240,242,244,245,255,4000,4094
mtu 9000
bridge-pvid 242
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2,3,10,30,40,41,71,90,100,240,242,244,245,255,4000,4094
mtu 9000
bridge-pvid 242
post-up ip link set dev vmbr0 promisc on
bridge-ageing 0
auto vlan242
iface vlan242 inet static
address 192.168.242.30/24
gateway 192.168.242.1
mtu 1500
vlan-raw-device vmbr1
dns-nameservers 192.168.40.53 192.168.41.53 192.168.242.1
dns-search mgmt.vegantechdude.com
auto cluster4094
iface cluster4094 inet static
address 192.168.255.3/29
mtu 9000
vlan-id 4094
vlan-raw-device vmbr0
auto vlan240
iface vlan240 inet static
address 192.168.240.30/24
mtu 1500
vlan-raw-device vmbr0
auto nas4000
iface nas4000 inet static
address 10.40.0.30/24
mtu 9000
vlan-id 4000
vlan-raw-device vmbr0
auto vlan41
iface vlan41 inet static
address 192.168.41.230/24
vlan-raw-device vmbr0The LXC container details:
Code:
arch: amd64
cores: 1
features: keyctl=1,nesting=1
hostname: ads-b
memory: 512
nameserver: 192.168.41.53 192.168.40.53
net0: name=eth41,bridge=vmbr0,gw=192.168.41.1,hwaddr=BC:24:11:35:62:EB,ip=192.168.41.90/24,mtu=1500,tag=41,type=veth
net1: name=eth40,bridge=vmbr1,hwaddr=BC:24:11:83:E3:39,ip=192.168.40.99/24,mtu=1500,tag=40,type=veth
onboot: 1
ostype: debian
rootfs: containers:vm-1090-disk-1,size=3G
swap: 0
tags: 41.90;community-script;os
unprivileged: 1From the rest of my network I can ping 192.168.240.30(my lan), 192.168.242.30, and 10.40.0.30(from a VM on another node on vlan4000)
This leads me to think my bridge is functioning correctly and vlan aware however I cannot ping 192.168.41.90 from anything except the hypervisor.
So far only one of the nodes has the connectx card, the rest of my nodes are still running 10g intel x710 cards until I get my 40g switch setup(currently I have a mikrotik switch which has 2 40g ports and the rest of the hosts are connected via its 10g ports)
brctl details
Code:
brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.2e49fb9b8d49 no bond0
veth1090i0
vmbr1 8000.244bfe4513f8 no enp6s0
veth1090i1
vmbr41v41 8000.000000000000 no
Code:
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.14.11-4-pve
Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: enp1s0v0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
Slave Interface: enp1s0f1d1
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: e2:f3:fa:25:9b:2a
Slave queue ID: 0
Slave Interface: enp1s0v0
MII Status: up
Speed: 40000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 2e:49:fb:9b:8d:49
Slave queue ID: 0if I put the LXC on my other bridge which is the 2.5g intel nic on the computer everything works.
Maybe I'm not searching for the right thing but I can't find any other topics that seem to address this issue, as visible I have tried to disable vlan offloading without effect as well and from my understanding should not be needed since I'm specifying all the vlan's I want and they are well under the card limit(this was already required by the intel x710)
If I can provide anything else to help I'd be happy to.
Thank you in advance.