[SOLVED] Network suddenly dropped

[TreiRei]

Hello,

One of my newly deployed proxmox servers suddenly went completely dark, while nobody was logged on it or any changes happening. When checking on it via KVM everything looked fine, but network was completely down.

Checking the syslog I notice the following at the exact time the monitoring noticed the server go down:

Aug 17 00:51:51 nox kernel: [55025.997764] ixgbe 0000:03:00.0: Adapter removed
Aug 17 00:51:53 nox systemd-networkd[1299]: enp3s0f0: Lost carrier
Aug 17 00:51:53 nox kernel: [55028.036686] vmbr0: port 1(enp3s0f0) entered disabled state

VE: 5.2-7
Kernel: 4.15.18-1-pve

Any idea what might have happened?
Thanks.

 

[TreiRei]

After yesterday's incident I wiped the system, made a clean fresh install of proxmox stable version, and the problem still persists.

Symptoms: everything works fine on the server for hours, then all of the sudden the network goes down, and server can only be reached via IPMI. Rebooting the server brings the network back up.

Any advice would be greatly appreciated. Let me know where to look for more info or what should I try to do the next time this happens.

Here's more info, as I've seen provided in other threads.

pveversion -v

proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.15.18-1-pve: 4.15.18-15
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.44-1-pve: 4.4.44-84
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

network config

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto vmbr1
iface vmbr1 inet manual
    address 10.0.1.1/24
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0

auto vmbr0
iface vmbr0 inet static
    address x.x.x.x
    netmask 255.255.255.0
    network x.x.x.0
    broadcast x.x.x.255
    gateway x.x.x.254
    dns-nameservers 8.8.8.8 8.8.4.4
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

network controllers

03:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

modinfo ixgbe

filename:       /lib/modules/4.15.18-1-pve/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version:        5.1.0-k
license:        GPL
description:    Intel(R) 10 Gigabit PCI Express Network Driver
author:         Intel Corporation, <linux.nics@intel.com>

syslog

Aug 18 02:44:25 nox kernel: [27917.781488] ixgbe 0000:03:00.0: Adapter removed
Aug 18 02:44:28 nox kernel: [27919.821553] vmbr0: port 1(eth0) entered disabled state

ip a (this is taken when the interface is functioning properly, I'll try to get a sample from when it goes down as well, next time it happens)

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 0c:c4:7a:6e:85:02 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:6e:85:03 brd ff:ff:ff:ff:ff:ff
4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:06:ac:e5:df:ff brd ff:ff:ff:ff:ff:ff
    inet6 fe80::54e9:8ff:fe78:fadc/64 scope link
       valid_lft forever preferred_lft forever
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:6e:85:02 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/24 brd x.x.x.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe6e:8502/64 scope link
       valid_lft forever preferred_lft forever
6: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 5a:21:88:59:42:5c brd ff:ff:ff:ff:ff:ff
7: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether fe:06:ac:e5:df:ff brd ff:ff:ff:ff:ff:ff
8: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 66:c9:21:a3:6f:42 brd ff:ff:ff:ff:ff:ff
9: tap102i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 6a:b7:16:2c:99:1e brd ff:ff:ff:ff:ff:ff

 

[TreiRei]

For anyone who might bump into a similar issue in the future, here's the solution in my case: the motherboard was faulty, support tech replaced it and problem seems to be fixed now.