Glass not working.

[Rex]

Good morning!

I'm completely green as it comes to glass. So I ordered the next hardware, hoping I understood enough from the info I gathered all over the place.

2 NIC's:
Intel X520-DA1, 82599EN (https://www.amazon.nl/-/en/dp/B01GHTXHMK)
Intel X520-DA2, 82599ES (https://www.amazon.nl/-/en/dp/B01DCZCA3O)

I inserted these 2 transceivers:
Intel E10GSFPSR Compatible SFP+ SR Multimode Transceiver, 10GBase-SR SFP+ Fibre Module, Dual LC Connector, 850nm, 300-Meter (https://www.amazon.nl/-/en/dp/B08L8RNRVJ)

Fiber:
LC to LC Fiber Optic Patch Cable 15m, OM3 Leads Multimode Duplex 50/125 (https://www.amazon.nl/-/en/dp/B088LVVP1D)

Every seems to work. Drivers installed fine. I can give them an IP address. Leds are lit. If I pull the wire on one side, both sides the lights go off and on again after I plug it back in. So they seem to see each other. The proxmox webinterface looks fine too. That's why I could create a VMBR10 from a fresh enp5s0f0 without any problem.

However, I can't transport even one little bit over that wire. Whatever I try.

This is what I see at the Proxmox prompt.

vmbr10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:b7:85:20:9a:88 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 scope global vmbr10
    valid_lft forever preferred_lft forever
    inet6 fe80::9ab7:85ff:fe20:9a88/64 scope link
    valid_lft forever preferred_lft forever

and this is what I see right now. My workstation:

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:b7:85:20:b6:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.99/8 brd 10.255.255.255 scope global noprefixroute enp2s0
    valid_lft forever preferred_lft forever
    inet6 fe80::1773:3065:8fe7:7de6/64 scope link noprefixroute
    valid_lft forever preferred_lft forever

ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

I can't ping anything.

$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
From 10.0.0.99 icmp_seq=1 Destination Host Unreachable
From 10.0.0.99 icmp_seq=2 Destination Host Unreachable
From 10.0.0.99 icmp_seq=3 Destination Host Unreachable
^C
--- 10.0.0.2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3061ms
pipe 4

The only thing that is suspicious is the boot time. Since the 2 cards are almost the same, one would expect about the same boot-times. But the server-side with 2 ports, a Gigabyte H370 HD** is booting fairly fast. My Asrock Z370 Killer SLI with the single port NIC takes forever to boot. At least 2 minutes. I know Intel can hold up the boot significant. So I'm not sure if this is normal. It is showing BIOS options first, (press F11 for boot, DEL for BIOS etc), than 2 lines text:

Initializing Intel(R) Boot Agent XE v2.4.32
PXE 2.1 Build 092 (WfM 2.0)

where it stays 2 minutes, than the BIOS again for about 10 seconds and after a while showing a black screen, it boots up normally without any problem. But why not both computers? Why only the Asrock? I can chooze Debian 12 and Windows 10 from there. Both have the same problem.

I have no clue why this is not working. It should imho.

Anyone? I'm totally desperado after about 10 hours trying and asking anyone I know. And I have no clue where to ask elsewhere either. Thanks in advance!

 

[twei]

Hey, can you provide the full "ip a" and "ip r s" output for both the PVE Server and the Workstation? The network configuration (can be found at "/etc/network/interfaces") of the Server would also be helpful

 

[Rex]

Problem is solved. I almost dare not write down what the problem was. Let me stress anyway that port 1 in a multiport nic does not necessarily have to be closest the motherboard. Even though all your other cards, in the last decade, in all your other systems are! Even de 1GB NIC in this very box has the first port closest to the mainboard. So I didn't give it even a second thought. Fortunately my rear was saved by a guy with the nick iam_tj on te Matrix.

Thanks for your reply, twei!

Let's hope this helps someone after all

 

[mrgohin]

Hello,

Your problem is very simply explained.

Your workstation (ws) is in a different subnet then your server:

10.0.0.2/24 != 10.0.0.99/8

You have to fix your ws subnet mask to 10.0.0.99/24 | 255.255.255.0

None the less I am more confused that you call fiber glass. Please stick to the term fiber so everyone knows what you mean.

As well there is not much difference for the computer if it has to send data out via copper or fiber. They are both layer 1 and handled in hardware whereas your problem was on layer 3 and software.

BR