Network card stops working

[jgracio]

Hello
I'm using Linux proxmox 4.15.18-21-pve #1 SMP PVE 4.15.18-48 on an Intel NUC for more than 2 years without problems.
For the last two months the NUC network card, after being working apparently smoothly, stops working not responding from any virtual network card. After restarting the OS or NUC by using the reset button, the network interface restarts working smoothly some days after it fails again.

Last time I connected a screen to NUC and got a photo from the errors that were scrolling.

Any idea how can I troubleshoot this intermittent error?

Thanks

 

[jgracio]

HELP please
It has failed again and I have no clue how to identify the problem

 

[mira]

Are you running the latest PVE 5 version? If not, try updating.

PVE 5 is EOL since July [0]. You should consider upgrading to the latest PVE 6 version [1].

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_frequently_asked_questions_2
[1] https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0

 

[jgracio]

Yes, @mira , I'm using the last version of PVE 5. I was planning to upgrade to version 6 but think is first better to fix the network problem. Meanwhile, the network interface continues to stop randomly(apparently).

 

[mikoz]

Hi, it seems like I'm experiencing a similar issue. My PVE instance seems to randomly lose all connection (the lights for the NIC port turn off). I've observed that this happens probably a few hours after I shutdown one of my containers (Minecraft server). The only way to restore the connection is to reboot the server. When I leave the container running everything works just fine without any issues.

PVE version: 6.0.7
Network interfaces log:

auto lo
iface lo inet loopback

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.60
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

I would appreciate some pointers. Thanks.

I wasn't sure whether to create a new thread for this, so apologies...

 

[mira]

Yes, @mira , I'm using the last version of PVE 5. I was planning to upgrade to version 6 but think is first better to fix the network problem. Meanwhile, the network interface continues to stop randomly(apparently).

Looks like you're not running the latest kernel version (4.15.18-30-pve). If you've already installed it, reboot and boot the latest one.

 

[mira]

Hi, it seems like I'm experiencing a similar issue. My PVE instance seems to randomly lose all connection (the lights for the NIC port turn off). I've observed that this happens probably a few hours after I shutdown one of my containers (Minecraft server). The only way to restore the connection is to reboot the server. When I leave the container running everything works just fine without any issues.

PVE version: 6.0.7
Network interfaces log:

auto lo
iface lo inet loopback

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.60
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

I would appreciate some pointers. Thanks.

I wasn't sure whether to create a new thread for this, so apologies...

First please update to the latest version (6.3) and provide the syslog/journal for that time period (a few minutes before until a few minutes after you lost the network connection).

 

[jgracio]

In my case, so far I'm not linking the network "failure" with any other event or fact. The NIC Just stops working with indication led blinking, being necessary to restart it.

I don't know if it is a configuration problem or a hardware problem. If I buy a new NUC and reinstall the same software the problem gets solved?
I'm lost

Looks like you're not running the latest kernel version (4.15.18-30-pve). If you've already installed it, reboot and boot the latest one.

Thanks again @mira
Please check the Summary of my node on attached photo, where I have my configuration with kernel version 4.15.18-30-pve. After this instalation my system have already booted several times.

Package versions
proxmox-ve: 5.4-2 (running kernel: 4.15.18-30-pve)
pve-manager: 5.4-15 (running version: 5.4-15/d0ec33c6)
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-56
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-42
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-56
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[mira]

What is the output of ethtool -i eth1 as well as ethtool -k eth1 ?

 

[jgracio]

What is the output of ethtool -i eth1 as well as ethtool -k eth1 ?

Thanks @mira

The output is:
root@proxmox:~# ethtool -i eno1
driver: e1000e
version: 3.4.1.1-NAPI
firmware-version: 0.13-4
expansion-rom-version:
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

root@proxmox:~# ethtool -k eno1
Features for eno1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]


However, ashamedly, I have to say there were no more failures over the last few days after replacing half-meter ethernet cable from the switch to the NUC (I believe this it was the fix).

 

[mikoz]

First please update to the latest version (6.3) and provide the syslog/journal for that time period (a few minutes before until a few minutes after you lost the network connection).

I have enabled the persistent journal logs and updated to the latest version (6.3.3) and the issue seems to have gone away. I have noticed that PVE is performing a package database update every night around the time when the issue used to occur.



I'll keep an eye on this.