Hello
we have a lot of this log on our Cisco switch,
our configuration is:
We have 11 PVE Cluster with 6 node for cluster.
each node have the same configuration:
HPE DL360 G10 2 x Xeon Silver 4210
HP InfiniBand FDR/Ethernet 40Gb 2-port 544+FLR-QSFP Adapte
PVE 8.2.2 with Linux Kernel 6.5.13-6 [but we have try also with newest Kernel, same problem, we use this kernel due to stability]
openvswitch-switch: 3.1.0-2+deb12u1
We use CEPH with nvme disk,.
we use DAC 40g Cable.
that are connected to 2 different switch cisco Nexus3000 C3164PQ Chassis (Nexus 9000 Series) on 2 differente rack with VPC
All port are configurated on the same way.
[switch 1]
interface port-channel8
description LINK TO SERVER
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
vpc 8
interface Ethernet1/8
description LINK #1 Po8
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
channel-group 8 mode active
[switch 2]
interface port-channel8
description LINK TO SERVER
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
vpc 8
interface Ethernet1/40
description LINK #2 Po8
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
channel-group 8 mode active
on both switch i get this on the log:
[switch 1]
2024 Dec 3 09:50:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 34 LLFC pause frames received on Ethernet1/8
2024 Dec 3 09:55:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 98 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:00:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 44 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:05:37 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 22 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:10:38 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 28 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:15:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 100 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:20:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 24 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:25:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 12 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:30:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 64 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:35:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 80 LLFC pause frames received on Ethernet1/8
[switch 2]
2024 Dec 3 09:49:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 09:54:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 146 LLFC pause frames received on Ethernet1/40
2024 Dec 3 09:59:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 64 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:04:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:09:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 32 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:14:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 96 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:19:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:24:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 16 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:29:47 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 34 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:34:47 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 130 LLFC pause frames received on Ethernet1/40
root@cl3kvm1:/home/davanzo# ethtool eno5
Settings for eno5:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
56000baseCR4/Full
56000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 40000baseCR4/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: yes
This pause fram could be a problem?
how we can solve this?
thanks.
we have a lot of this log on our Cisco switch,
our configuration is:
We have 11 PVE Cluster with 6 node for cluster.
each node have the same configuration:
HPE DL360 G10 2 x Xeon Silver 4210
HP InfiniBand FDR/Ethernet 40Gb 2-port 544+FLR-QSFP Adapte
PVE 8.2.2 with Linux Kernel 6.5.13-6 [but we have try also with newest Kernel, same problem, we use this kernel due to stability]
openvswitch-switch: 3.1.0-2+deb12u1
We use CEPH with nvme disk,.
we use DAC 40g Cable.
that are connected to 2 different switch cisco Nexus3000 C3164PQ Chassis (Nexus 9000 Series) on 2 differente rack with VPC
All port are configurated on the same way.
[switch 1]
interface port-channel8
description LINK TO SERVER
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
vpc 8
interface Ethernet1/8
description LINK #1 Po8
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
channel-group 8 mode active
[switch 2]
interface port-channel8
description LINK TO SERVER
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
vpc 8
interface Ethernet1/40
description LINK #2 Po8
switchport
switchport mode trunk
switchport trunk allowed vlan 2-4094
spanning-tree port type edge trunk
mtu 9216
channel-group 8 mode active
on both switch i get this on the log:
[switch 1]
2024 Dec 3 09:50:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 34 LLFC pause frames received on Ethernet1/8
2024 Dec 3 09:55:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 98 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:00:36 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 44 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:05:37 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 22 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:10:38 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 28 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:15:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 100 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:20:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 24 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:25:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 12 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:30:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 64 LLFC pause frames received on Ethernet1/8
2024 Dec 3 10:35:39 dc1-leaf-5 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 80 LLFC pause frames received on Ethernet1/8
[switch 2]
2024 Dec 3 09:49:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 09:54:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 146 LLFC pause frames received on Ethernet1/40
2024 Dec 3 09:59:45 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 64 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:04:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:09:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 32 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:14:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 96 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:19:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 20 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:24:46 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 16 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:29:47 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 34 LLFC pause frames received on Ethernet1/40
2024 Dec 3 10:34:47 dc1-leaf-6 %-SLOT1-2-BCM_SYSLOG_LLFC_PAUSE_FRAME: 130 LLFC pause frames received on Ethernet1/40
root@cl3kvm1:/home/davanzo# ethtool eno5
Settings for eno5:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
56000baseCR4/Full
56000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseKX4/Full
40000baseCR4/Full
40000baseSR4/Full
1000baseX/Full
10000baseCR/Full
10000baseSR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 40000baseCR4/Full
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 40000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: yes
This pause fram could be a problem?
how we can solve this?
thanks.