Problem

[Dominik2000]

Hello,

I have a problem with one of my Proxmox instances. I have one HP DL380 Gen6 server with Proxmox 6-1.8 (Hercules) and one custom server with an Asus mainboard and Intel CPU also with Proxmox 6-1.8 (Whiskey). I want to connect these two over network with zfs over iscsi. I bought two IBM QLE8142-IBMX Dual Port 10 Gbit/s SFP+ SR CNA Server Adapter. On both servers the cards are detected, the qlge driver is used and I have two new ethernet ports. I have configured both with static ip. On Whiskey there is no problem, the link goes up. On Hercules the link never goes up.
I have switched the network cards, because I was worried that one will be damaged, but after the switch the card in Whiskey works, and in Hercules not.

I have attached screenshots, if you need more informations let me know.

Hercules pveversion -v:

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-4.15.18-26-pve: 4.15.18-54
pve-kernel-4.13.4-1-pve: 4.13.4-26
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Whiskey pveversion -v:

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-22
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Thank you in advance
Dominik

 

[LnxBil]

How are the two cards connected to each other?

 

[Dominik2000]

I have conncted both ports, to test different cables. One is this: https://www.amazon.de/gp/product/B07HDHYS9M/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1
and one is this:
https://www.amazon.de/gp/product/B06XKGFWHV/ref=ppx_yo_dt_b_asin_title_o05_s00?ie=UTF8&psc=1

 

[LnxBil]

So, two twinax cables between the two servers and ons reports up and the other not? That is strange indeed.

Please install ethtool (apt-get install -qq ethtool) and run ethtool ens1f0 on both servers (also change f0 with f1).

 

[Dominik2000]

On hercules:

root@hercules:~# ethtool ens1f0
Settings for ens1f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x000060f7 (24823)
drv probe link ifdown ifup rx_err tx_err hw wol
Link detected: no
root@hercules:~# ethtool ens1f1
Settings for ens1f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x000060f7 (24823)
drv probe link ifdown ifup rx_err tx_err hw wol
Link detected: no

On whiskey:

root@whiskey:~# ethtool ens4f0
Settings for ens4f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x000060f7 (24823)
drv probe link ifdown ifup rx_err tx_err hw wol
Link detected: yes
root@whiskey:~# ethtool ens4f1
Settings for ens4f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x000060f7 (24823)
drv probe link ifdown ifup rx_err tx_err hw wol
Link detected: yes

I don't understand why one machine has no link detected on both ports. I have another fibre cable, should I try this?

 

[LnxBil]

I have another fibre cable, should I try this?

Yes, go on. Try also connecting only one cable.

 

[Dominik2000]

Sadly also with the fibre cable no success.

I read that this IBM QLogic Controller only works with IBM Fibre Cables and Transceivers, so I ordered one, and it will be delivered today. So maybe this works, I hope so!

Dominik

 

[LnxBil]

I read that this IBM QLogic Controller only works with IBM Fibre Cables and Transceivers, so I ordered one, and it will be delivered today. So maybe this works, I hope so!

Yeah, vendor locking ... cool stuff!

 

[Dominik2000]

I don't get it to work... I have antoher pair of HP NC523SFP maybe I get them to work, when I update the firmware.