[SOLVED] Problems with networking on an HP DL360 G10 and Proxmox 6.3.1

[Nick_W]

I've got a brand new DL360 G10, 2 x Xeon G 5218, plenty of ram, and installed the latest Proxmox 6.3.1 on it without issue. No errors or hiccups during install. Networking however has been a problem since the start. The machine had a 4 port Gigabit ethernet card on board, which connected (picked up a DHCP address), and worked while setting up disk arrays. As soon as the system booted into Debian, the interfaces stopped working. No connectivity on the switch, not even lights for the physical layer. As I had a riser, and a 10GB 2 port Intel x540-T2 card, I decided to give that a go. This card appeared immediately, and in Debian connects on the PHY layer, as well as ending up in the '/etc/network/interfaces' as the card configured for the bridge, with the IP address I set during install. However, even this setup networking isn't functioning, I can neither ping the machine IP, or anything on the network from the machine. I decided to retry the install, and made sure to select that card during the networking step (has an odd name enpos1r4u4 or something similar), and set it up with the same address as before. No change.

This isn't the first proxmox system I've set up, I have another DL 360 G10, running 6.2-15, without any issue, and have used proxmox for 5 or 6 years without ever having issues like this.

I can probably get outputs from commands, but due to the pandemic I'm working from home, and this machine is in an office a fair distance from me. I think I will try and install 6.2 next, unless anyone thinks this problem is something simple to resolve?

Thanks, Nick

 

[mira]

If you can somehow get them, the logs from a boot (journalctl -b or /var/log/syslog*) would be great, as well as the output of lspci -nnk.

 

[Nick_W]

Alright, here's lspic -nnk



and the journal files, probably going to try a 6.2 install, just to see if it is the hardware..

 

[Nick_W]

Anything odd in those files?
So, I installed 6.2, and got basically back to the same situation.

One thing I've noticed is that the devices (card and bridge) were in LOWER UP, vmbr0 is in state UP, but enp1s0f4u4 is state UNKNOWN, I've seen some people saying it was an issue with PCIe ports, so I ensured it was on an x8, unfortunately, ethtool isn't installed, so I can't try and manually set the speed. Also 'ip link set enp1s0f4u4 up' doesn't throw errors, but also doesn't change the UNKNOWN state.

It connects on a LINK level to both 10G and 1G ports, with cat 6E cable, it seems to have the right driver ixgbe, I've tried removing everything else on the PCIe ports to see if it was a conflict there, but no luck.

 

[mira]

In the lspci output there is no mention of any other NIC than the Intel one. Which kind of onboard NIC is there? According to the journal it seems to be a USB NIC?

As the Intel NIC seems to be up and running, could you provide the network config (/etc/network/interfaces) as well as the output of ip -details a? And the output of lsusb -vv.

 

[Nick_W]

The onboard NICs are some sort of PCI based connector, the call it a FlexibleLOM, it has 4 GBE interfaces. Something like this

Here are those files..

 

[mira]

In both the journal and the lsusb output you can see a virtual iLO NIC [0] which you're trying to use. If you take a look the ip output, you can see that there are 4 enoX interfaces, 2 ensX interfaces and the virtual NIC which has 'u4' in its name (indicating USB [1]).
Try to configure the enoX ones as those are the onboard ones or the ensX ones as those are the Intel NIC you installed.

[0] https://support.hpe.com/hpesc/public/docDisplay?docId=a00104665en_us&docLocale=en_US
[1] https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html

 

[Nick_W]

I had removed the flexibleLOM one, as it was just cluttering up the network output, I have added it back in, not much changed.

 

[Nick_W]

In both the journal and the lsusb output you can see a virtual iLO NIC [0] which you're trying to use. If you take a look the ip output, you can see that there are 4 enoX interfaces, 2 ensX interfaces and the virtual NIC which has 'u4' in its name (indicating USB [1]).
Try to configure the enoX ones as those are the onboard ones or the ensX ones as those are the Intel NIC you installed.

[0] https://support.hpe.com/hpesc/public/docDisplay?docId=a00104665en_us&docLocale=en_US
[1] https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html

Oh, so the ensX ones are the ones I need to configure, thanks!

 

[Nick_W]

Finally, still not sure why I wasn't able to get the onboard ones working, but setting it up with the enS ones did the trick, thanks!