Fresh v8 install. Only initial NIC is working

S

SCSJeff

New Member
Sep 14, 2023
6
0
1
Hello all,

New user to Proxmox. Just did a fresh install from the latest ISO to a Dell PE R715 w/ the 4 Broadcom LOMs and an add-on 4-port Intel NIC.

I have network connectivity to the host via the 1st configured NIC during install (lom1/eno1). The web UI shows all the NICs. However, after troubleshooting why my VM wasn't communicating, I discovered that none of the other NICs are working...

The other 3 Broadcom LOMs give me a link light when plugging into the switch. But, "ip a" shows status "down"
The Intel NIC ports don't even give a link light when I connect to the switch????

Hoping for some help here...

Thank you,

Jeff

Here's some of my config results:

lspci | grep 'Ethernet'

Code: 
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
25:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
25:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
26:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
26:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02

cat /etc/network/interfaces:

Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

iface enp37s0f0 inet manual

iface enp37s0f1 inet manual

iface enp38s0f0 inet manual

iface enp38s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
#VM Traffic

auto vmbr0
iface vmbr0 inet static
        address 10.47.10.91/24
        gateway 10.47.10.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

ip a:

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether f0:4d:a2:40:93:bb brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
    link/ether f0:4d:a2:40:93:bd brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether f0:4d:a2:40:93:bf brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether f0:4d:a2:40:93:bf brd ff:ff:ff:ff:ff:ff permaddr f0:4d:a2:40:93:c1
    altname enp2s0f1
6: enp37s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1b:21:3b:1d:00 brd ff:ff:ff:ff:ff:ff
7: enp37s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1b:21:3b:1d:01 brd ff:ff:ff:ff:ff:ff
8: enp38s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1b:21:3b:1d:04 brd ff:ff:ff:ff:ff:ff
9: enp38s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1b:21:3b:1d:05 brd ff:ff:ff:ff:ff:ff
10: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether f0:4d:a2:40:93:bf brd ff:ff:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:4d:a2:40:93:bb brd ff:ff:ff:ff:ff:ff
    inet 10.47.10.91/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::f24d:a2ff:fe40:93bb/64 scope link
       valid_lft forever preferred_lft forever
12: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether f0:4d:a2:40:93:bd brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f24d:a2ff:fe40:93bd/64 scope link
       valid_lft forever preferred_lft forever

eno2 is what I'm currently testing with and have a link light

ethtool eno2:

Code: 
Settings for eno2:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: no
 
Additional Info:

I have a 2nd identical server that is exhibiting the same exact issue as well...
 
Wow... what a bummer. No help :(

I had previously tested XCP-NG on these servers without issue. I "thought" I was going to like Proxmox better so had all but decided to go the Proxmox route. I guess I'm going back to XCP-NG.
 
The other 3 Broadcom LOMs give me a link light when plugging into the switch. But, "ip a" shows status "down"
Click to expand...

seem that it's missing "auto ..." for theses interfaces.

(the "autostart" checkbox on the gui when you are editing nic)

The Intel NIC ports don't even give a link light when I connect to the switch????
Click to expand...
that's strange. Do you have a link light at server physical boot ?
 
spirit said:
(the "autostart" checkbox on the gui when you are editing nic)
Click to expand...
I had tried both ways. Initially, I was using the bond0 and it appears to leave the individual member NICs autostart off. But, yes, I tried enabling that to no evail...
spirit said:
Do you have a link light at server physical boot ?
Click to expand...
Nope (oh wait, you mean at the initial boot before the OS kicks in? I'll check)


Thank you for the reply :)
 
OK, no link lights on initial boot (but I think that is normal for the add-in cards). I booted off of an Ubuntu Live USB and the link lights on the Intel NIC work fine and the installer showed the NIC(port 1) as connected.

So, I'm assuming a driver issue (at least for the Intel card) in Proxmox. I have no idea why the broadcom LOM is only working for the 1st port ???

In that case, I really don't want to spend any more time troubleshooting this. I need something that is going to work out of the box and be reliable for all my Dell PowerEdge servers :(
 
Update: I decided to give one last go before wiping the server out and re-installing XCP-NG...

I set ALL interfaces to AutoStart (except for eno1, apparently that comes up automatically regardless) AND Rebooted the server.

Now all interfaces are giving a link light and showing as UP in Proxmox as I plug cables in!

Not sure if all interfaces have to be set to autostart to get them to work or if the reboot of the server did the trick (or both)?

I'm going to go back and test disabling autostart on the interfaces I don't need and see what happens...
 
You must log in or register to reply here.
Top Bottom