No Network after Host boot

[waynieack]

I set up my first Proxmox ve host running version 3.4-3 with kernel 3.10.0-5 (I'm using this kernel in order to do VGA pass through). When I boot the host with the new kernel (3.10.0-5), I have no network connectivity from the host or guests. In order to get connectivity back, I have to run the CMDs below and reboot any guest that automatically started on boot. I don't have this issue with kernel 2.6.32-37.

modprobe -rf igb
modprobe -rf igbvf
modprobe igb
service networking restart



I scrubbed through the dmesg log and pulled out anything that had to do with the network:

<snip>
[ 1.584625] Intel(R) Gigabit Ethernet Network Driver - version 5.2.9.4
[ 1.584995] Copyright (c) 2007-2014 Intel Corporation.
[ 1.585341] libata version 3.00 loaded.
[ 1.602984] Areca RAID Controller0: Model ARC-1225, F/W V1.52 2014-09-11
[ 1.603617] scsi0 : Areca SAS/SATA RAID Controller (RAID6 capable)
[ 1.603617] arcmsr version v1.30.0X.19-20140509
[ 1.603617]
[ 1.604617] arcmsr 0000:06:00.0: irq 72 for MSI/MSI-X
[ 1.604623] arcmsr 0000:06:00.0: irq 73 for MSI/MSI-X
[ 1.604629] arcmsr 0000:06:00.0: irq 74 for MSI/MSI-X
[ 1.604635] arcmsr 0000:06:00.0: irq 75 for MSI/MSI-X
[ 1.604670] arcmsr0: msi-x enabled
[ 1.605652] igb: 0000:04:00.0: igb_validate_option: max_vfs - SR-IOV VF devices set to 7
[ 1.606364] ahci 0000:00:11.4: version 3.0
[ 1.606458] ahci 0000:00:11.4: irq 76 for MSI/MSI-X
[ 1.606510] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[ 1.607162] ahci 0000:00:11.4: flags: 64bit ncq pm led clo pio slum part ems apst
[ 1.707110] pci 0000:04:10.0: [8086:1520] type 00 class 0x020000
[ 1.708546] pci 0000:04:10.4: [8086:1520] type 00 class 0x020000
[ 1.709886] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.0.2-k
[ 1.709930] pci 0000:04:11.0: [8086:1520] type 00 class 0x020000
[ 1.710524] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 1.711570] igbvf 0000:04:11.0: enabling device (0000 -> 0002)
[ 1.712113] igbvf 0000:04:11.0: irq 78 for MSI/MSI-X
[ 1.712120] igbvf 0000:04:11.0: irq 79 for MSI/MSI-X
[ 1.712130] igbvf 0000:04:11.0: irq 80 for MSI/MSI-X
[ 1.716974] igbvf 0000:04:11.0: PF still in reset state. Is the PF interface up?
[ 1.717595] igbvf 0000:04:11.0: Assigning random MAC address.
[ 1.722718] igbvf 0000:04:11.0: PF still resetting
[ 1.723179] igbvf 0000:04:11.0: Intel(R) I350 Virtual Function
[ 1.723526] igbvf 0000:04:11.0: Address: da:7d:54:79:38:87
[ 1.724106] pci 0000:04:11.4: [8086:1520] type 00 class 0x020000
[ 1.725255] igbvf 0000:04:11.4: enabling device (0000 -> 0002)
[ 1.725753] igbvf 0000:04:11.4: irq 81 for MSI/MSI-X
[ 1.725759] igbvf 0000:04:11.4: irq 82 for MSI/MSI-X
[ 1.725770] igbvf 0000:04:11.4: irq 83 for MSI/MSI-X
[ 1.730638] igbvf 0000:04:11.4: PF still in reset state. Is the PF interface up?
[ 1.731260] igbvf 0000:04:11.4: Assigning random MAC address.
[ 1.736361] igbvf 0000:04:11.4: PF still resetting
[ 1.736787] igbvf 0000:04:11.4: Intel(R) I350 Virtual Function
[ 1.737137] igbvf 0000:04:11.4: Address: 66:dc:e3:9b:1d:3d
[ 1.737708] pci 0000:04:12.0: [8086:1520] type 00 class 0x020000
[ 1.738776] igbvf 0000:04:12.0: enabling device (0000 -> 0002)
[ 1.739342] igbvf 0000:04:12.0: irq 84 for MSI/MSI-X
[ 1.739347] igbvf 0000:04:12.0: irq 85 for MSI/MSI-X
[ 1.739359] igbvf 0000:04:12.0: irq 86 for MSI/MSI-X
[ 1.744200] igbvf 0000:04:12.0: PF still in reset state. Is the PF interface up?
[ 1.744817] igbvf 0000:04:12.0: Assigning random MAC address.
[ 1.749920] igbvf 0000:04:12.0: PF still resetting
[ 1.750350] igbvf 0000:04:12.0: Intel(R) I350 Virtual Function
[ 1.750711] igbvf 0000:04:12.0: Address: 1e:b3:74:b1:25:99
[ 1.751226] pci 0000:04:12.4: [8086:1520] type 00 class 0x020000
[ 1.752175] igbvf 0000:04:12.4: enabling device (0000 -> 0002)
[ 1.752672] igbvf 0000:04:12.4: irq 87 for MSI/MSI-X
[ 1.752678] igbvf 0000:04:12.4: irq 88 for MSI/MSI-X
[ 1.752690] igbvf 0000:04:12.4: irq 89 for MSI/MSI-X
[ 1.757503] igbvf 0000:04:12.4: PF still in reset state. Is the PF interface up?
[ 1.758123] igbvf 0000:04:12.4: Assigning random MAC address.
[ 1.762007] usb 1-1: new high-speed USB device number 2 using ehci-pci
[ 1.763249] igbvf 0000:04:12.4: PF still resetting
[ 1.763670] igbvf 0000:04:12.4: Intel(R) I350 Virtual Function
[ 1.764020] igbvf 0000:04:12.4: Address: 2a:f4:8b:fb:7e:fb
[ 1.764528] pci 0000:04:13.0: [8086:1520] type 00 class 0x020000
[ 1.765282] igbvf 0000:04:13.0: enabling device (0000 -> 0002)
[ 1.765802] igbvf 0000:04:13.0: irq 90 for MSI/MSI-X
[ 1.765808] igbvf 0000:04:13.0: irq 91 for MSI/MSI-X
[ 1.765819] igbvf 0000:04:13.0: irq 92 for MSI/MSI-X
[ 1.770661] igbvf 0000:04:13.0: PF still in reset state. Is the PF interface up?
[ 1.771283] igbvf 0000:04:13.0: Assigning random MAC address.
[ 1.776387] igbvf 0000:04:13.0: PF still resetting
[ 1.776809] igbvf 0000:04:13.0: Intel(R) I350 Virtual Function
[ 1.777162] igbvf 0000:04:13.0: Address: 96:6d:6d:77:b3:d5
[ 1.777802] igb 0000:04:00.0: irq 93 for MSI/MSI-X
[ 1.777808] igb 0000:04:00.0: irq 94 for MSI/MSI-X
[ 1.861552] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 1.861904] igb 0000:04:00.0: eth5: (PCIe:5.0GT/s:Width x2)
[ 1.862282] igb 0000:04:00.0 eth5: MAC: 54:a0:50:86:34:f7
[ 1.862745] igb 0000:04:00.0: eth5: PBA No: 106300-000
[ 1.865100] igb 0000:04:00.0: LRO is disabled
[ 1.865452] igb 0000:04:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 1.867348] igb 0000:04:00.1: irq 95 for MSI/MSI-X
[ 1.867356] igb 0000:04:00.1: irq 96 for MSI/MSI-X
[ 1.941608] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 1.941610] igb 0000:04:00.1: eth6: (PCIe:5.0GT/s:Width x2)
[ 1.941611] igb 0000:04:00.1 eth6: MAC: 54:a0:50:86:34:f8
[ 1.941729] igb 0000:04:00.1: eth6: PBA No: 106300-000
[ 1.943079] ata5: SATA link down (SStatus 0 SControl 300)
[ 1.943453] ata7: SATA link down (SStatus 0 SControl 300)
[ 1.944161] igb 0000:04:00.1: LRO is disabled
[ 1.944525] igb 0000:04:00.1: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
[ 1.945277] igbvf 0000:04:10.0: enabling device (0000 -> 0002)
[ 1.945839] igbvf 0000:04:10.0: irq 97 for MSI/MSI-X
[ 1.945845] igbvf 0000:04:10.0: irq 98 for MSI/MSI-X
[ 1.945856] igbvf 0000:04:10.0: irq 99 for MSI/MSI-X
[ 1.950718] igbvf 0000:04:10.0: PF still in reset state. Is the PF interface up?
[ 1.951346] igbvf 0000:04:10.0: Assigning random MAC address.
[ 1.956445] igbvf 0000:04:10.0: PF still resetting
[ 1.956884] igbvf 0000:04:10.0: Intel(R) I350 Virtual Function
[ 1.957239] igbvf 0000:04:10.0: Address: 7e:db:1c:a3:9d:1a
[ 1.957709] igbvf 0000:04:10.4: enabling device (0000 -> 0002)
[ 1.958248] igbvf 0000:04:10.4: irq 100 for MSI/MSI-X
[ 1.958254] igbvf 0000:04:10.4: irq 101 for MSI/MSI-X
[ 1.958267] igbvf 0000:04:10.4: irq 102 for MSI/MSI-X
[ 1.963225] igbvf 0000:04:10.4: PF still in reset state. Is the PF interface up?
[ 1.963844] igbvf 0000:04:10.4: Assigning random MAC address.
[ 1.968969] igbvf 0000:04:10.4: PF still resetting
[ 1.969402] igbvf 0000:04:10.4: Intel(R) I350 Virtual Function
[ 1.969748] igbvf 0000:04:10.4: Address: 8a:34:7f:ea:e4:d3
<snip\>

<snip>
[ 14.937445] IPMI System Interface driver.
[ 14.937809] ipmi_si: probing via ACPI
[ 14.938158] ipmi_si 00:0a: [io 0x0ca2] regsize 1 spacing 1 irq 0
[ 14.938498] ipmi_si: Adding ACPI-specified kcs state machine
[ 14.938908] ipmi_si: probing via SMBIOS
[ 14.939244] ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
[ 14.939587] ipmi_si: Adding SMBIOS-specified kcs state machine duplicate interface
[ 14.940258] ipmi_si: probing via SPMI
[ 14.940596] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0
[ 14.940935] ipmi_si: Adding SPMI-specified kcs state machine duplicate interface
[ 14.941607] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca2, slave address 0x0, irq 0
[ 15.168440] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000428-0x000000000000042f (\GPE0) (20130517/utaddress-254)
[ 15.170189] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 15.171251] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x000000000000052c-0x000000000000052d (\GPIV) (20130517/utaddress-254)
[ 15.172994] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 15.174068] lpc_ich: Resource conflict(s) found affecting gpio_ich
[ 15.219748] ipmi_si 00:0a: Found new BMC (man_id: 0x001043, prod_id: 0x0e73, dev_id: 0x20)
[ 15.220369] ipmi_si 00:0a: IPMI kcs interface initialized
[ 15.229633] mei_me 0000:00:16.0: Device doesn't have valid ME Interface
[ 15.229980] mei_me 0000:00:16.0: initialization failed.
[ 15.385974] iTCO_vendor_support: vendor-support=0
[ 15.390170] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.10
[ 15.390533] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
[ 15.415127] [drm] Initialized drm 1.1.0 20060810
[ 15.441278] [drm] AST 2300 detected
[ 15.441627] [drm] dram 1632000000 6 16 02000000
[ 15.441997] [TTM] Zone kernel: Available graphics memory: 8201514 kiB
[ 15.442341] [TTM] Zone dma32: Available graphics memory: 2097152 kiB
[ 15.442683] [TTM] Initializing pool allocator
[ 15.443025] [TTM] Initializing DMA pool allocator
[ 154.632179] tun: Universal TUN/TAP device driver, 1.6
[ 154.633131] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 154.721435] Netfilter messages via NETLINK v0.30.
[ 155.986032] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 156.148778] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 156.426258] ip_set: protocol 6
[ 161.715618] device tap102i0 entered promiscuous mode
[ 161.719508] vmbr0: port 2(tap102i0) entered forwarding state
[ 161.719956] vmbr0: port 2(tap102i0) entered forwarding state
[ 161.720478] IPv6: ADDRCONF(NETDEV_CHANGE): vmbr0: link becomes ready
[ 166.024948] device tap103i0 entered promiscuous mode
[ 166.028959] vmbr0: port 3(tap103i0) entered forwarding state
[ 166.029397] vmbr0: port 3(tap103i0) entered forwarding state
[ 169.960534] kvm: zapping shadow pages for mmio generation wraparound
[ 170.121394] device tap104i0 entered promiscuous mode
[ 170.125360] vmbr0: port 4(tap104i0) entered forwarding state
[ 170.125883] vmbr0: port 4(tap104i0) entered forwarding state
[ 200.200810] kvm: zapping shadow pages for mmio generation wraparound
<snip\>

 

[waynieack]

The problem seems to be a bug with udev. There's a patch for it, but I wasn't able to successfully apply the patch.

With this problem you will see about a 1 min delay on boot and after the delay you will see an error that looks something like:

udev: error changing net interface name rename7 to eth1: file exists
udev: error changing net interface name rename6 to eth0: file exists

Here are a link related to the problem and the patch:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1090002
https://bugs.centos.org/view.php?id=7591


I figured out a work around by moving the 2 physical nics to eth10 and eth11 in the udev rules after the virtual functions. Moving eth0 and eth1 to eth10 and eth11 may also work if you don't have virtual functions, I haven't tested it though.

vi /etc/udev/rules.d/70-persistent-net.rules

# Virtual Functions
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="d6:ae:b4:73:e5:5d", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="46:d9:00:cc:1a:c6", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="2a:49:0d:2b:8b:80", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="ce:de:92:ca:61:0a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a6:3c:15:70:26:c2", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:20:a1:64:13:25", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth5"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="86:52:93:77:ce:40", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth6"

# Physical Interfaces
# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.2/0000:04:00.0 (igb)
[B]SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="54:a0:50:86:34:f7", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="[COLOR=#ff0000]eth10[/COLOR]"[/B]

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.2/0000:04:00.1 (igb)
[B]SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="54:a0:50:86:34:f8", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="[COLOR=#ff0000]eth11[/COLOR][/B]"

Special thanks to everyone who helped me with this problem.