Bond IEEE 802.3ad Dynamic link aggregation kein Lastenausgleich

K

Kundo

Member
Feb 11, 2023
9
0
6
Hallo,
ich habe auf zwei Proxmox VE Server Bond IEEE 802.3ad Dynamic link aggregation eingerichtet und bekomme max. nur 2,5 Gbit Datenübertragung.

LACP = aktiviert
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Netzwerkkabel ordnungsgemäß verbunden
Ausfallschutz funktioniert
Switch zeigt 5 Gbit bei Trunk1 und Trunk2

Wenn ich iperf3 ausführe, werden mir 2.35 Gbits/sec angezeigt.

Auch wenn ich große Daten zwischen den beiden Serveren (die Backups) kopiere, komme ich bei 2,5 Gbit ohne Bond auf die gleiche Zeit, wie mit Bond. Der einzige Unterschied, ist kann wärenden des kopierens immer ein LAN Kabel im Bond abklemmen, was zeigt, dass der Ausfallschutz zumindest funktioniert.

nano /etc/network/interfaces

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address 192.168.2.200/24
#onBoard 1Gbit

auto enp5s0
iface enp5s0 inet manual
mtu 9000
#PCI-e 2,5Gbit

auto enp6s0
iface enp6s0 inet manual
mtu 9000
#PCI-e 2,5Gbit

auto bond0
iface bond0 inet manual
bond-slaves enp5s0 enp6s0
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-lacp-rate fast
mtu 9000
#PCI-e 2,5Gbit

auto vmbr1
iface vmbr1 inet static
address 192.168.1.200/24
gateway 192.168.1.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
mtu 9000
#bond LACP

Auf beiden Servern die gleiche Einstellung, entsprechend der IP Adressen angepasst.

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v6.8.12-13-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 1c:fd:08:7d:ba:1a
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 11
Partner Key: 1000
Partner Mac Address: 1c:2a:a3:24:28:4f

Slave Interface: enp5s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 1c:fd:08:7d:ba:1a
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 1c:fd:08:7d:ba:1a
port key: 11
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: 1c:2a:a3:24:28:4f
oper key: 1000
port priority: 32768
port number: 2
port state: 61

Slave Interface: enp6s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 1c:fd:08:7d:ba:1b
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 1c:fd:08:7d:ba:1a
port key: 11
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: 1c:2a:a3:24:28:4f
oper key: 1000
port priority: 32768
port number: 1
port state: 61



Ethernet Channel Bonding Driver: v6.8.12-13-pve

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 1c:86:0b:35:67:0e
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 11
Partner Key: 1001
Partner Mac Address: 1c:2a:a3:24:28:4f

Slave Interface: enp3s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 1c:86:0b:35:67:0e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 1c:86:0b:35:67:0e
port key: 11
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: 1c:2a:a3:24:28:4f
oper key: 1001
port priority: 32768
port number: 3
port state: 61

Slave Interface: enp4s0
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:de:80:b7:80:4e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 1c:86:0b:35:67:0e
port key: 11
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 32768
system mac address: 1c:2a:a3:24:28:4f
oper key: 1001
port priority: 32768
port number: 4
port state: 61

iperf3 -c 192.168.1.200 (auch auf den anderen Server)
1753894055063.png
 
dmesg |grep bond
[85260.566357] vmbr1: port 1(bond0) entered disabled state
[85260.567930] bond0: (slave enp6s0): Releasing backup interface
[85260.581590] bond0: (slave enp5s0): Removing an active aggregator
[85260.581642] bond0: (slave enp5s0): Releasing backup interface
[85260.598853] vmbr1: port 1(bond0) entered blocking state
[85260.598864] vmbr1: port 1(bond0) entered forwarding state
[85260.789826] bond0: (slave enp5s0): Enslaving as an active interface with a down link
[85260.989900] bond0: (slave enp6s0): Enslaving as an active interface with a down link
[85261.601355] vmbr1: port 1(bond0) entered disabled state
[85264.136391] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[85264.136439] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[85264.136443] bond0: active interface up!
[85264.136452] vmbr1: port 1(bond0) entered blocking state
[85264.136460] vmbr1: port 1(bond0) entered forwarding state
[85765.052221] vmbr1: port 1(bond0) entered disabled state
[85765.053300] bond0: (slave enp6s0): Releasing backup interface
[85765.068590] bond0: (slave enp5s0): Releasing backup interface
[85765.086483] vmbr1: port 1(bond0) entered blocking state
[85765.086496] vmbr1: port 1(bond0) entered forwarding state
[85765.276696] bond0: (slave enp5s0): Enslaving as a backup interface with a down link
[85765.473687] bond0: (slave enp6s0): Enslaving as a backup interface with a down link
[85766.102078] vmbr1: port 1(bond0) entered disabled state
[85768.620994] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[85768.620998] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[85768.621003] bond0: active interface up!
[85768.621013] vmbr1: port 1(bond0) entered blocking state
[85768.621023] vmbr1: port 1(bond0) entered forwarding state
[85768.725051] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[86030.820554] vmbr1: port 1(bond0) entered disabled state
[86030.822013] vmbr1: port 1(bond0) entered blocking state
[86030.822023] vmbr1: port 1(bond0) entered forwarding state
[86135.128689] vmbr1: port 1(bond0) entered disabled state
[86135.130177] vmbr1: port 1(bond0) entered blocking state
[86135.130187] vmbr1: port 1(bond0) entered forwarding state
[86889.953791] bond0: (slave enp6s0): speed changed to 0 on port 2
[86889.972349] bond0: (slave enp5s0): speed changed to 0 on port 1
[86890.052077] bond0: (slave enp5s0): link status definitely down, disabling slave
[86890.052186] bond0: (slave enp6s0): link status definitely down, disabling slave
[86890.052189] bond0: now running without any active interface!
[86890.052220] vmbr1: port 1(bond0) entered disabled state
[86893.171990] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[86893.171996] bond0: active interface up!
[86893.172007] vmbr1: port 1(bond0) entered blocking state
[86893.172017] vmbr1: port 1(bond0) entered forwarding state
[86893.275958] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[87462.592055] bond0: (slave enp6s0): speed changed to 0 on port 2
[87462.610611] bond0: (slave enp5s0): speed changed to 0 on port 1
[87462.663292] bond0: (slave enp5s0): link status definitely down, disabling slave
[87462.663402] bond0: (slave enp6s0): link status definitely down, disabling slave
[87462.663405] bond0: now running without any active interface!
[87462.663467] vmbr1: port 1(bond0) entered disabled state
[87465.888233] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[87465.888280] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[87465.888284] bond0: active interface up!
[87465.888295] vmbr1: port 1(bond0) entered blocking state
[87465.888305] vmbr1: port 1(bond0) entered forwarding state
[87499.760886] bond0: (slave enp6s0): speed changed to 0 on port 2
[87499.778810] bond0: (slave enp5s0): speed changed to 0 on port 1
[87499.791511] bond0: (slave enp5s0): link status definitely down, disabling slave
[87499.791630] bond0: (slave enp6s0): link status definitely down, disabling slave
[87499.791633] bond0: now running without any active interface!
[87499.791665] vmbr1: port 1(bond0) entered disabled state
[87503.014404] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[87503.014416] bond0: active interface up!
[87503.014426] vmbr1: port 1(bond0) entered blocking state
[87503.014436] vmbr1: port 1(bond0) entered forwarding state
[87503.118404] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[87535.914616] bond0: (slave enp6s0): speed changed to 0 on port 2
[87535.932955] bond0: (slave enp5s0): speed changed to 0 on port 1
[87535.981730] bond0: (slave enp5s0): link status definitely down, disabling slave
[87535.981842] bond0: (slave enp6s0): link status definitely down, disabling slave
[87535.981845] bond0: now running without any active interface!
[87535.981868] vmbr1: port 1(bond0) entered disabled state
[87539.206544] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[87539.206590] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[87539.206594] bond0: active interface up!
[87539.206605] vmbr1: port 1(bond0) entered blocking state
[87539.206616] vmbr1: port 1(bond0) entered forwarding state
[88333.433265] bond0: (slave enp5s0): speed changed to 0 on port 1
[88333.443936] bond0: (slave enp5s0): link status definitely down, disabling slave
[88333.443945] bond0: active interface up!
[88333.550236] bond0: (slave enp6s0): speed changed to 0 on port 2
[88333.551926] bond0: (slave enp6s0): link status definitely down, disabling slave
[88333.551936] bond0: now running without any active interface!
[88333.551983] vmbr1: port 1(bond0) entered disabled state
[88336.988748] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[88336.988759] bond0: active interface up!
[88336.988769] vmbr1: port 1(bond0) entered blocking state
[88336.988777] vmbr1: port 1(bond0) entered forwarding state
[88337.092749] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[90732.152490] vmbr1: port 1(bond0) entered disabled state
[90732.153980] vmbr1: port 1(bond0) entered blocking state
[90732.153989] vmbr1: port 1(bond0) entered forwarding state
[90812.049616] vmbr1: port 1(bond0) entered disabled state
[90812.051152] vmbr1: port 1(bond0) entered blocking state
[90812.051162] vmbr1: port 1(bond0) entered forwarding state
[134027.770624] bond0: (slave enp6s0): speed changed to 0 on port 2
[134027.779564] bond0: (slave enp5s0): speed changed to 0 on port 1
[134027.834195] bond0: (slave enp5s0): link status definitely down, disabling slave
[134027.834307] bond0: (slave enp6s0): link status definitely down, disabling slave
[134027.834311] bond0: now running without any active interface!
[134027.836236] vmbr1: port 1(bond0) entered disabled state
[134058.201490] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[134058.201529] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[134058.201533] bond0: active interface up!
[134058.201544] vmbr1: port 1(bond0) entered blocking state
[134058.201554] vmbr1: port 1(bond0) entered forwarding state
[137507.327564] vmbr1: port 1(bond0) entered disabled state
[137507.328974] bond0: (slave enp6s0): Releasing backup interface
[137507.348225] bond0: (slave enp5s0): Removing an active aggregator
[137507.348327] bond0: (slave enp5s0): Releasing backup interface
[137507.364406] vmbr1: port 1(bond0) entered blocking state
[137507.364418] vmbr1: port 1(bond0) entered forwarding state
[137507.557236] bond0: (slave enp5s0): Enslaving as an active interface with a down link
[137507.757260] bond0: (slave enp6s0): Enslaving as an active interface with a down link
[137508.372653] vmbr1: port 1(bond0) entered disabled state
[137510.901665] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[137510.901699] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[137510.901703] bond0: active interface up!
[137510.901713] vmbr1: port 1(bond0) entered blocking state
[137510.901721] vmbr1: port 1(bond0) entered forwarding state
[147569.536435] vmbr1: port 1(bond0) entered disabled state
[147569.537534] bond0: (slave enp5s0): Releasing backup interface
[147569.537538] bond0: (slave enp5s0): the permanent HWaddr of slave - 1c:fd:08:7d:ba:1a - is still in use by bond - set the HWaddr of slave to a different address to avoid conflicts
[147569.555032] bond0: (slave enp6s0): Releasing backup interface
[147569.572082] vmbr1: port 1(bond0) entered blocking state
[147569.572092] vmbr1: port 1(bond0) entered forwarding state
[147569.763336] bond0: (slave enp5s0): Enslaving as an active interface with a down link
[147569.961386] bond0: (slave enp6s0): Enslaving as an active interface with a down link
[147570.610949] vmbr1: port 1(bond0) entered disabled state
[147573.106840] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[147573.106845] bond0: (slave enp5s0): making interface the new active one
[147573.106981] bond0: active interface up!
[147573.106992] vmbr1: port 1(bond0) entered blocking state
[147573.107001] vmbr1: port 1(bond0) entered forwarding state
[147573.210809] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[148163.011518] vmbr1: port 1(bond0) entered disabled state
[148163.012510] bond0: (slave enp5s0): Releasing active interface
[148163.012514] bond0: (slave enp5s0): the permanent HWaddr of slave - 1c:fd:08:7d:ba:1a - is still in use by bond - set the HWaddr of slave to a different address to avoid conflicts
[148163.012583] bond0: (slave enp6s0): making interface the new active one
[148163.031968] bond0: (slave enp6s0): Releasing active interface
[148163.054768] vmbr1: port 1(bond0) entered blocking state
[148163.054778] vmbr1: port 1(bond0) entered forwarding state
[148163.243301] bond0: (slave enp5s0): Enslaving as a backup interface with a down link
[148163.441256] bond0: (slave enp6s0): Enslaving as a backup interface with a down link
[148164.069622] vmbr1: port 1(bond0) entered disabled state
[148166.693543] bond0: (slave enp5s0): link status definitely up, 2500 Mbps full duplex
[148166.693549] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[148166.693583] bond0: (slave enp6s0): link status definitely up, 2500 Mbps full duplex
[148166.693586] bond0: active interface up!
[148166.693597] vmbr1: port 1(bond0) entered blocking state
[148166.693607] vmbr1: port 1(bond0) entered forwarding state
[253917.888066] vmbr1: port 1(bond0) entered disabled state
[253917.891918] vmbr1: port 1(bond0) entered blocking state
[253917.891932] vmbr1: port 1(bond0) entered forwarding state
[315689.506375] vmbr1: port 1(bond0) entered disabled state
[315689.507806] vmbr1: port 1(bond0) entered blocking state
[315689.507815] vmbr1: port 1(bond0) entered forwarding state


dmesg |grep bond
[ 7.079098] bond0: (slave enp3s0): Enslaving as a backup interface with a down link
[ 7.222187] bond0: (slave enp4s0): Enslaving as a backup interface with a down link
[ 7.251673] vmbr1: port 1(bond0) entered blocking state
[ 7.251677] vmbr1: port 1(bond0) entered disabled state
[ 7.251685] bond0: entered allmulticast mode
[ 7.252021] bond0: entered promiscuous mode
[ 10.377453] bond0: (slave enp3s0): link status definitely up, 2500 Mbps full duplex
[ 10.377475] bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond
[ 10.377481] bond0: active interface up!
[ 10.377504] vmbr1: port 1(bond0) entered blocking state
[ 10.377506] vmbr1: port 1(bond0) entered forwarding state
[ 10.793453] bond0: (slave enp4s0): link status definitely up, 2500 Mbps full duplex

Verstehe ich da etwas mit dem Lastenausgleich falsch oder passt die Konfigurierung nicht?

Danke.
 
news said:
Moin, läuft doch alles Super bei Dir! Mehr kann man mit einer Verbindung nicht erwarten.
Click to expand...
Wenn ich das mit dem LACP in älteren Videos mit dem 1 Gbit Netzwerk sehe, dann kopieren die dort ihre Daten mit 112 Mb/s ohne LACP und mit LACP kommen die auf 220+ Mb/s, also Lastenausgleich. Ich müsste ja dann auf die Geschwindigkeit eines 5 Gbit Netzwerks kommen, oder etwa nicht?
 
dann haben sie aber mehr als nur 1 datenstrom. ein datenstrom kann niemals die kapazität eines netzwerkinterfaces bei lacp überschreiten. wenn du dagegen 2, 3, 5, 10 datenströme (von unterschiedlichen quellen/zielen) hast, dann verteilen die sich auch auf die verschiedenen interfaces.

beispiel: 2x iperf von zwei unterschiedlichen computern zum selben server sollte zusammen die kapazität eines 2.5gbit interfaces überschreiten.
 
Da ist ein Denkfehler, 1 Verbindung dann auch nur die einfache Geschwindigkeit.
Ich habe das auch alles in meinem Netzwerk mit 3 NIC, 2x 1 GBit/s, 1x 2.5 GBit/s pro PC durch und getestet.
Da ich bei meinem Nas (Proxmox VE, Proxmox BS) mit ZFS und HDD, SDD die 2.5 GBit/s ausschöpfen kann, reicht mir dies für mein Backup-Netzwerk aus.
Über die 1 GBit/s Adapter läuft dann das management Netzwerk..
 
Je nach Lastausgleich Modus brauchst du z.B. bei Layer 2+3 unterschiedliche IPs und bei Layer 3+4 reichen schon verschiedene Ports auf einer IP. Aber wie schon mehrfach geschrieben wurde, 1 Datenstrom läuft immer auf 1 Interface.
 
Okay, wenn ich das jetzt richtig verstanden habe, werden aus zwei 2,5 Gbit Netzwerkkarten eine Virtuelle 2,5 Gbit Netzwerkkarte und keine 5 Gbit, mit zwei Leitungen für den Ausfallschutz.
Ich dachte, dass die Pakete über beide Leitungen aufgeteilt werden und sich dadurch die Übertragung erhöht, so wie beim SDRAM > DDRRAM...

Danke.
 
Es werden Pakete aufgeteilt, aber nicht für einen Stream. Im Regelfall hast du verschiedene VMs oder verschiedene Dienste, die auf verschiedenen Ports sprechen und dann hast du beide Links gesättigt. Auch wenn 2 Clients auf einen Server zugreifen, sind das 2 Streams, die aufgeteilt werden.
 
  • Like
Reactions: Kundo and beisser
You must log in or register to reply here.
Top Bottom