[SOLVED] hp 10Gb nics in full meshed network strange slow performance

[silvered.dragon]

Hi friends,
I have 3 node ceph cluster with identical server and configuration.
The ceph network is made by dual 10Gb HP 533FLR nics in a full meshed network.
with command Iperf

- from node1 to every node and from node2 to every node,
I always obtain a maximum throughput of 6,70 Gb/s.
Instead
- from node3 to every node I always got 9.9Gb/s.
same hardware
same servers
same proxmox
same twinax sfp+ copper cables
jambo frame enabled on every nic
cannot figure out where is the difference beetween node3 and the others.

 

[silvered.dragon]

Only difference I found was with lspci | egrep -i --color 'network|ethernet'
on node 3 I got

03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)
03:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)

on other nodes

03:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
03:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)

 

[silvered.dragon]

solved:
sorry for the useless post but maybe it can be nice to know how I fixed this - in BIOS properties you have to setup QPI resources to maximum performance and in system power management setup to maximum consumption.
In bios POST you will notice 8 GT/s QPI before was 6,4.