PVE-Server not reachable after installation

[dmox]

Hi there,
I installed my new hardware from Thomas-Krenn with PVE (v.5.2.1).
Now after the installation has finished everything seems fine and the message to open the webinterface in the browser appears with the set up IP address.
I am in the same network as the PVE-Server, but unfortunately I am not able to reach the host, neither via a ping or via webinterface.

The server has a NIC with two ports (40 GBit/s) and they are named with enp96s0f[0/1]. I read in the Network Model documentation that there are some naming conventions (only eth[0-99] allowed), but I am pretty sure this is rather not the problem here.

There seems to be something wrong, but atm I do not have a clue (shame on me). Any suggestions? What information could I serve you that would support the debugging?

Thank you very much for any hint.

 

[dcsapak]

can you post your /etc/network/interfaces and the output of systemctl status pveproxy ?

 

[dmox]

Hi dcsapak,
thank you for your response.
The desired output is as follows.

cat /etc/network/interfaces


systemctl status pveproxy.service


UPDATE
I did some further testing to shed some light on this issue:

• I eliminated the network side, so if I plug in the network cable in another machine, it works as is should.
• I tested some live linux distros and all of them had problems establishing a link. E.g. Ubuntu displayed both network ports but they were shown as unplugged although the cable was plugged in.

 

[dcsapak]

maybe you have plugged in the wrong interface?

 

[dmox]

maybe you have plugged in the wrong interface?

I only have two options (enp96s0f0 or enp96s0f1). Both result in the same. I am currently talking to Thomas-Krenn because this might be a simple and unlucky case of delivered broken hardware.

Do you have any other tip how I could test the functionality?

Sincerely, dmox

 

[dcsapak]

what does

ip addr
ip link
ip route

say?

also, any relevant messages in 'dmesg' ?

 

[dmox]

ip addr

ip link

ip route

also, any relevant messages in 'dmesg' ?

I could not see any valuable information provided by dmesg except:

 

[Stoiko Ivanov]

The NO-CARRIER tag in the output of enp96s0f0 (and as a result also in the output for vmbr0) tells you that no link is detected (otherwise it would say LOWER_UP).

* Either it is just plugged into the wrong interface (just try plugging the cable into the other interface)
* Or the driver does not recognize that there is a link (dmesg could provide some information here)
* you could/should also check the other side of the cable (switch/router , if possible just plug in a linux box and check the output)

 

[dmox]

The NO-CARRIER tag in the output of enp96s0f0 (and as a result also in the output for vmbr0) tells you that no link is detected (otherwise it would say LOWER_UP).

* Either it is just plugged into the wrong interface (just try plugging the cable into the other interface)
* Or the driver does not recognize that there is a link (dmesg could provide some information here)
* you could/should also check the other side of the cable (switch/router , if possible just plug in a linux box and check the output)

Thank you for your response.

• I get what you mean with the wrong interface -> only enp96s0f0 is a bridge_port for vmbr0. And if I plug it in enp96s0f1 it will therefore not work. => I'll check this one tomorrow (although I thought I've checked the other interface as well...)
  
• Do you know what to search for in the output of dmesg to check if the driver does not recognize the plugged in cable?
• I already tried some live-linux'es and they behave similar to proxmo (nevertheless, if I failed with just pluggin it in the wrong interface, then this will not work of course...)

 

[Stoiko Ivanov]

I get what you mean with the wrong interface -> only enp96s0f0 is a bridge_port for vmbr0. And if I plug it in enp96s0f1 it will therefore not work. => I'll check this one tomorrow (although I thought I've checked the other interface as well...)

You could set the link manually up on enp96s0f1 by running `ip link set enp96s0f1 up` - and then check with `ip link` if the LOWER_UP flag appears.
You could also open a shell and leave `ip monitor` run in it while replugging cables - that should show some output if link states change

check `dmesg` before and after replugging, not all drivers do write messages if a link state changes.

The idea of checking the other side, was meant to check the switch, where the proxmox host is connected, whether it sees a link

the output of `ethtool enp96s0f0` (and `ethtool enp96s0f1`) could also provide you with some insights

 

[dmox]

You could set the link manually up on enp96s0f1 by running `ip link set enp96s0f1 up` - and then check with `ip link` if the LOWER_UP flag appears.
You could also open a shell and leave `ip monitor` run in it while replugging cables - that should show some output if link states change

check `dmesg` before and after replugging, not all drivers do write messages if a link state changes.

The idea of checking the other side, was meant to check the switch, where the proxmox host is connected, whether it sees a link

the output of `ethtool enp96s0f0` (and `ethtool enp96s0f1`) could also provide you with some insights

Hi Stoiko, I set enp96s0f0 manually up and checked ip link afterwards, which resulted in the following:

When I did the same for enp96s0f1 it also got the NO-CARRIER flag.
I also opened another shell and run ip monitor which gave me these messages (a new one appears about every 2 secs):

The link on the side of the switch does not show any activity, similar to when no hardware is plugged in.
Unfortunately I do not have ethtool installed, can I catch this up via an offline way?

 

[Stoiko Ivanov]

If both the switch side, and the linux side tell you that there's no link - at least they agree - maybe check the cables?

you could also try removing the kernel module (`lspci -nnk` should tell you which one is used) by using `rmmod i40e` (if i40e is the module), and afterwards load it again with `modprobe i40e`, and checking the output of `dmesg`

 

[dmox]

If both the switch side, and the linux side tell you that there's no link - at least they agree - maybe check the cables?

you could also try removing the kernel module (`lspci -nnk` should tell you which one is used) by using `rmmod i40e` (if i40e is the module), and afterwards load it again with `modprobe i40e`, and checking the output of `dmesg`

Hey, thanks again for your reply. Of course, I already checked the cable (plugging it in another machine results in a proper network functionality).
After removing and adding the kernel module i40e, dmesg shows the following regarding this module:

Thanks again for your willingness to help me. Currently, I am pretty helpless and about to RMA the hardware. Especially, because Thomas-Krenn delivers those as pre-tested PVE-Boxes and therefore they should work out of the box.

 

[Stoiko Ivanov]

the dmesg output doesn't suggest any problems there (apart from the `Link not ready` indicating that there is no link) - maybe try to connect the port directly to another computer and see if anything changes. Also the output of ethtool might give you some suggestions.

 

[dmox]

the dmesg output doesn't suggest any problems there (apart from the `Link not ready` indicating that there is no link) - maybe try to connect the port directly to another computer and see if anything changes. Also the output of ethtool might give you some suggestions.

I already connected the port to a laptop and it works as it should. As I mentioned, ethtool is not installed and I can not catch this up via apt. Do you know another way doing so, like downloading the sources/pre-compiled binaries from another pc and then copying them onto the PVE-Server?

 

[Stoiko Ivanov]

if the connection between a laptop and your new server works - then I would guess that the problem lies with the switch?

the ethtool debian package should install cleanly (no other dependencies apart from libc6) - you could just download it (e.g. from http://ftp.at.debian.org/debian/pool/main/e/ethtool/ethtool_4.8-1+b1_amd64.deb) and copy it onto the box via usb flashdrive and install it via `dpkg -i`

 

[dmox]

if the connection between a laptop and your new server works - then I would guess that the problem lies with the switch?

the ethtool debian package should install cleanly (no other dependencies apart from libc6) - you could just download it (e.g. from http://ftp.at.debian.org/debian/pool/main/e/ethtool/ethtool_4.8-1+b1_amd64.deb) and copy it onto the box via usb flashdrive and install it via `dpkg -i`

I got you wrong. I did not try a cross-over connection between my laptop and the server. I do not own such a cable...
Thank you for the advice. I thought this would be more complicated (in the meantime I played around with apt-offline). I'll try this tomorrow.

 

[dmox]

Good morning,
I installed ethtool and run it on both interfaces. The results are as follows:

@Stoiko Ivanov can you see something noteworthy here?

There is one thing I'll test asap: the interfaces only support 1/10 GE (I always thought, the downward compatibility is until the bottom of 10 MBit/s) and the switch connected to it only offers speed up to 100 MBit/s. It might be the case, that the server simply can not establish a link because the minimum network interface speed is not served... (I never was confronted with such a case before).

UPDATE
I am really sorry for wasting your time: the upper block is exactly what was happening here. The switch (only supporting up to 100 MBit/s) could not serve the PVE-Server with a valid link, because the network interface of the server supports only 1/10 GE connections. I successfully tested the whole setup with a new switch that supports 10 GE.

 

[Stoiko Ivanov]

Ahh - jupp - bit me one time as well - the intel 10G nics only support 1G and 10G (not sure about others, but wouldn't be surprised if very few 10G nics supported 100M) - Glad the Problem has been identified!

 

[dmox]

Ahh - jupp - bit me one time as well - the intel 10G nics only support 1G and 10G (not sure about others, but wouldn't be surprised if very few 10G nics supported 100M) - Glad the Problem has been identified!

Yeah, I've learned something new today
Thank you very much again at this point, @Stoiko Ivanov, you helped me a lot!