Glass not working.

R

Rex

Member
Oct 22, 2022
17
5
8
Good morning!

I'm completely green as it comes to glass. So I ordered the next hardware, hoping I understood enough from the info I gathered all over the place.

2 NIC's:
Intel X520-DA1, 82599EN (https://www.amazon.nl/-/en/dp/B01GHTXHMK)
Intel X520-DA2, 82599ES (https://www.amazon.nl/-/en/dp/B01DCZCA3O)

I inserted these 2 transceivers:
Intel E10GSFPSR Compatible SFP+ SR Multimode Transceiver, 10GBase-SR SFP+ Fibre Module, Dual LC Connector, 850nm, 300-Meter (https://www.amazon.nl/-/en/dp/B08L8RNRVJ)

Fiber:
LC to LC Fiber Optic Patch Cable 15m, OM3 Leads Multimode Duplex 50/125 (https://www.amazon.nl/-/en/dp/B088LVVP1D)

Every seems to work. Drivers installed fine. I can give them an IP address. Leds are lit. If I pull the wire on one side, both sides the lights go off and on again after I plug it back in. So they seem to see each other. The proxmox webinterface looks fine too. That's why I could create a VMBR10 from a fresh enp5s0f0 without any problem.

However, I can't transport even one little bit over that wire. Whatever I try.

This is what I see at the Proxmox prompt.
Code: 
vmbr10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 98:b7:85:20:9a:88 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 scope global vmbr10
    valid_lft forever preferred_lft forever
    inet6 fe80::9ab7:85ff:fe20:9a88/64 scope link
    valid_lft forever preferred_lft forever

and this is what I see right now. My workstation:
Code: 
enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:b7:85:20:b6:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.99/8 brd 10.255.255.255 scope global noprefixroute enp2s0
    valid_lft forever preferred_lft forever
    inet6 fe80::1773:3065:8fe7:7de6/64 scope link noprefixroute
    valid_lft forever preferred_lft forever

Code: 
ethtool enp2s0
Settings for enp2s0:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes


I can't ping anything.
Code: 
$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
From 10.0.0.99 icmp_seq=1 Destination Host Unreachable
From 10.0.0.99 icmp_seq=2 Destination Host Unreachable
From 10.0.0.99 icmp_seq=3 Destination Host Unreachable
^C
--- 10.0.0.2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3061ms
pipe 4


The only thing that is suspicious is the boot time. Since the 2 cards are almost the same, one would expect about the same boot-times. But the server-side with 2 ports, a Gigabyte H370 HD** is booting fairly fast. My Asrock Z370 Killer SLI with the single port NIC takes forever to boot. At least 2 minutes. I know Intel can hold up the boot significant. So I'm not sure if this is normal. It is showing BIOS options first, (press F11 for boot, DEL for BIOS etc), than 2 lines text:

Initializing Intel(R) Boot Agent XE v2.4.32
PXE 2.1 Build 092 (WfM 2.0)

where it stays 2 minutes, than the BIOS again for about 10 seconds and after a while showing a black screen, it boots up normally without any problem. But why not both computers? Why only the Asrock? I can chooze Debian 12 and Windows 10 from there. Both have the same problem.

I have no clue why this is not working. It should imho.

Anyone? I'm totally desperado after about 10 hours trying and asking anyone I know. And I have no clue where to ask elsewhere either. Thanks in advance!
 
Last edited:
Hey, can you provide the full "ip a" and "ip r s" output for both the PVE Server and the Workstation? The network configuration (can be found at "/etc/network/interfaces") of the Server would also be helpful
 
  • Like
Reactions: Rex
Problem is solved. I almost dare not write down what the problem was. Let me stress anyway that port 1 in a multiport nic does not necessarily have to be closest the motherboard. Even though all your other cards, in the last decade, in all your other systems are! Even de 1GB NIC in this very box has the first port closest to the mainboard. So I didn't give it even a second thought. Fortunately my rear was saved by a guy with the nick iam_tj on te Matrix.

Thanks for your reply, twei!

Let's hope this helps someone after all ;)
 
Last edited:
Hello,

Your problem is very simply explained.

Your workstation (ws) is in a different subnet then your server:

10.0.0.2/24 != 10.0.0.99/8

You have to fix your ws subnet mask to 10.0.0.99/24 | 255.255.255.0

None the less I am more confused that you call fiber glass. Please stick to the term fiber so everyone knows what you mean.

As well there is not much difference for the computer if it has to send data out via copper or fiber. They are both layer 1 and handled in hardware whereas your problem was on layer 3 and software.

BR
 
Last edited:
mrgohin said:
Hello,

Your problem is very simply explained.

Your workstation (ws) is in a different subnet then your server:

10.0.0.2/24 != 10.0.0.99/8

You have to fix your ws subnet mask to 10.0.0.99/24 | 255.255.255.0

None the less I am more confused that you call fiber glass. Please stick to the term fiber so everyone knows what you mean.

As well there is not much difference for the computer if it has to send data out via copper or fiber. They are both layer 1 and handled in hardware whereas your problem was on layer 3 and software.

BR
Click to expand...

"Your workstation (ws) is in a different subnet then your server:"

No, it's not. The netmask may differ but they are both in 10.0.0.0/24. It isn't well configured. True that, but it works.

"None the less I am more confused that you call fiber glass. Please stick to the term fiber so everyone knows what you mean."

Everybody knows what I mean. You are the exception. And even you aren't. Ambiguity is impossible here. Besides, I already explained, I'm new in glass links. But even at my regular pub, they knew what I was talking about when I said I had laid glass at home. They are all clueless but impressed. "What do you need that for?" was a question that crossed the table. Besides, I wasn't here for lessons in English. Thanks anyway.

"As well there is not much difference for the computer if it has to send data out via copper or fiber. They are both layer 1 and handled in hardware whereas your problem was on layer 3 and software."

Again. No. The connector was in the wrong hole like I explained already. Something you must have missed while writing your comment.

Besides, there is a lot different between copper and glass. Not only in software but in hardware too! Everything can go wrong there and may add to problems you already have in software.

- Copper is not suitable for 10GB links if the cable is longer than 3 meters. (This is the short-handed version)
- The RJ45 (or copper) transceivers get hot and need passive cooling. Many switches can't handle them because of that.
- The RJ45 transceivers are way more expensive.
- Glass, (or fiber) has problems with dust and smudge.

And I'm sure I forgot about a whole lot of other relevant things.
 
Last edited:
...to be fair: most people in the English speaking world are calling it fiber (or fibre) as it technically isn't normal glass, that term is mainly used in the German-ish speaking world (i'll just count the Netherlands to that). Also there isn't really a difference for the OS whether you're sending data over fiber or copper since, as he rightfully stated, that is mostly handled in hardware. Other than that you're right, the subnet difference is meaningless and there are many things you have to consider with the physical side of fiber that just aren't a problem with copper. Glad you figured it out :)
 
twei said:
...to be fair: most people in the English speaking world are calling it fiber (or fibre) as it technically isn't normal glass, that term is mainly used in the German-ish speaking world (i'll just count the Netherlands to that).
Click to expand...

I'm from the Netherlands indeed. So English is not my native language. Yet I hope you can read me flawlessly.

We call fiber 'glasvezel'. Mostly shorted to glass. Some call it fiber. Anyway, everybody knows what you mean if you are talking about glass. Phrases like "Heb jij glas thuis liggen?" can only mean that you are talking about your internet connection. At least for most of us. Not many people are like me and pull fiberglass throughout the house. ;)
 
Last edited:
You must log in or register to reply here.
Top Bottom