NIC card/VM Stop working

James Richmond · Jan 22, 2019

I am not to sure what all information will be needed to help me resolve this issue so please ask and I will provide it.

Here is what is going on.

We have 3 node cluster on 3 R710 servers. All 3 servers have the same hardware with a total of 8 network ports. 4 ports on the motherboard and a 4 port intel NIC.

If I have any VM's that use any port on that Intel NIC then I cannot access that VM through the web gui. I can access it through the proxmox console. I good example is I have a test VM that is running Pfsense. I can tell it to use two virtual ports that are using the built in NIC of the R710 but if I switch just one port like the WAN port of pfsense to the intel NIC then I can no longer access it at all through the web gui (can still get to it through proxmox gui) till I remove that port. Now oddly enough this happens on all 3 servers. All 3 intel network cards are acting that way.

The only fix seems to wait a week or two (found that out by accident) or reboot all the nodes. Then that intel NIC will work again.

Please any advice would be great. Again I don't know what logs to provide as I don't see any issues anywhere. If you tell me what to provide I would be more than happy to.

Thanks again!

James Richmond · Jan 25, 2019

Still having the issue. Please someone!!!

Stoiko Ivanov · Jan 28, 2019

Please post the VM config.
Also what do you mean by "using the Intel NIC" - do you have a vmbr for each intel-NIC or do you try to do PCI-passthrough?

* the output of the network-configuration on the node `ip addr`, `ip link`, `ip route` would be helpful, as well as the output of the network configuration from inside the guest `ifconfig`, `route -n` for FreeBSD/pfsense

James Richmond · Jan 29, 2019

Stoiko Ivanov said:
Please post the VM config.
Also what do you mean by "using the Intel NIC" - do you have a vmbr for each intel-NIC or do you try to do PCI-passthrough?

* the output of the network-configuration on the node `ip addr`, `ip link`, `ip route` would be helpful, as well as the output of the network configuration from inside the guest `ifconfig`, `route -n` for FreeBSD/pfsense

When I am having the issue of not being able to get to the VM gui via vmbr2 I will switch the LAN to use vmbr0 and then change vmbr3 to vmbr0 as well to get it off that 4 port NIC card. Also when using that NIC card I can migrate that VM to any 1 of the 3 servers and it will not work. I have a 4 port intel NIC installed in all 3 servers.

I use vmbr's and not PCI-passthrough. Here is the config for the VM:

bootdisk: virtio0
cores: 4
ide2: none,media=cdrom
memory: 8192
name: Pfsense
net0: virtio=D2:77:45:7F:62

5,bridge=vmbr2
net1: virtio=F6:4F

E:28:02:19,bridge=vmbr3
net2: e1000=1A

A:C0:B2:A1:19,bridge=vmbr0
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=a06bfc90-99be-43d8-8a9e-5f586ba8f41a
sockets: 2
virtio0: RBS_vm:vm-110-disk-0,size=80G
vmgenid: b3d62739-66aa-4924-8a84-1a7eedb10e2d

And for the Node:
PVE1 Node:
auto lo
iface lo inet loopback

iface eno1 inet manual
#PVE

iface eno2 inet manual
#Ceph

iface eno3 inet manual
#CoroSync

iface eno4 inet manual

iface enp8s0f0 inet manual
#LAN

iface enp8s0f1 inet manual
#WAN

iface enp9s0f0 inet manual
#FreePBX

iface enp9s0f1 inet manual
#PBXFiber

auto vmbr0
iface vmbr0 inet static
address 192.168.0.251
netmask 255.255.255.0
gateway 192.168.0.250
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.10.10.1
netmask 255.255.255.0
bridge-ports eno2
bridge-stp off
bridge-fd 0
#CephBR

auto vmbr2
iface vmbr2 inet static
address 192.168.0.140
netmask 255.255.255.0
bridge-ports enp8s0f0
bridge-stp off
bridge-fd 0
#pfSenseLAN

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp8s0f1
bridge-stp off
bridge-fd 0
#pfSenseWAN

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp9s0f0
bridge-stp off
bridge-fd 0
#FreePBX

auto vmbr5
iface vmbr5 inet manual
bridge-ports enp9s0f1
bridge-stp off
bridge-fd 0
#PBXFiber

auto vmbr6
iface vmbr6 inet static
address 10.10.10.11
netmask 255.255.255.0
bridge-ports eno3
bridge-stp off
bridge-fd 0
#CoroSync

PVE2 Node:

auto lo
iface lo inet loopback

iface eno1 inet manual
#PVE

iface eno2 inet manual
#Ceph

iface eno3 inet manual
#CoroSync

iface eno4 inet manual

iface enp8s0f0 inet manual
#LAN

iface enp8s0f1 inet manual
#WAN

iface enp9s0f0 inet manual
#FreePBX

iface enp9s0f1 inet manual
#PBXFiber

auto vmbr0
iface vmbr0 inet static
address 192.168.0.252
netmask 255.255.255.0
gateway 192.168.0.250
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.10.10.2
netmask 255.255.255.0
bridge-ports eno2
bridge-stp off
bridge-fd 0
#CephBR

auto vmbr2
iface vmbr2 inet static
address 192.168.0.143
netmask 255.255.255.0
bridge-ports enp8s0f0
bridge-stp off
bridge-fd 0
#pfSenseLAN

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp8s0f1
bridge-stp off
bridge-fd 0
#pfSenseWAN

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp9s0f0
bridge-stp off
bridge-fd 0
#FreePBX

auto vmbr5
iface vmbr5 inet manual
bridge-ports enp9s0f1
bridge-stp off
bridge-fd 0
#PBXFiber

auto vmbr6
iface vmbr6 inet static
address 10.10.10.12
netmask 255.255.255.0
bridge-ports eno3
bridge-stp off
bridge-fd 0
#CoroSync

PVE3 Node:

auto lo
iface lo inet loopback

iface eno1 inet manual
#PVE

iface eno2 inet manual
#Ceph

iface eno3 inet manual
#CoroSync

iface eno4 inet manual

iface enp8s0f0 inet manual
#LAN

iface enp8s0f1 inet manual
#WAN

iface enp9s0f0 inet manual
#FreePBX

iface enp9s0f1 inet manual
#PBXFiber

auto vmbr0
iface vmbr0 inet static
address 192.168.0.253
netmask 255.255.255.0
gateway 192.168.0.250
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.10.10.3
netmask 255.255.255.0
bridge-ports eno2
bridge-stp off
bridge-fd 0
#CephBR

auto vmbr2
iface vmbr2 inet static
address 192.168.0.146
netmask 255.255.255.0
bridge-ports enp8s0f0
bridge-stp off
bridge-fd 0
#pfSenseLAN

auto vmbr3
iface vmbr3 inet manual
bridge-ports enp8s0f1
bridge-stp off
bridge-fd 0
#pfSenseWAN

auto vmbr4
iface vmbr4 inet manual
bridge-ports enp9s0f0
bridge-stp off
bridge-fd 0
#FreePBX

auto vmbr5
iface vmbr5 inet manual
bridge-ports enp9s0f1
bridge-stp off
bridge-fd 0
#PBXFiber

auto vmbr6
iface vmbr6 inet static
address 10.10.10.13
netmask 255.255.255.0
bridge-ports eno3
bridge-stp off
bridge-fd 0
#CoroSync

James Richmond · Jan 29, 2019

PVE1:

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:54 brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:56 brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:58 brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 84:2b:2b:0e:99:5a brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP group default qlen 1000

link/ether 00:1f:29:60:19:09 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP group default qlen 1000

link/ether 00:1f:29:60:19:08 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN group default qlen 1000

link/ether 00:1f:29:60:19:0b brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN group default qlen 1000

link/ether 00:1f:29:60:19:0a brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:54 brd ff:ff:ff:ff:ff:ff

inet 192.168.0.251/24 brd 192.168.0.255 scope global vmbr0

valid_lft forever preferred_lft forever

inet6 fe80::862b:2bff:fe0e:9954/64 scope link

valid_lft forever preferred_lft forever

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:56 brd ff:ff:ff:ff:ff:ff

inet 10.10.10.1/24 brd 10.10.10.255 scope global vmbr1

valid_lft forever preferred_lft forever

inet6 fe80::862b:2bff:fe0e:9956/64 scope link

valid_lft forever preferred_lft forever

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:1f:29:60:19:09 brd ff:ff:ff:ff:ff:ff

inet 192.168.0.140/24 brd 192.168.0.255 scope global vmbr2

valid_lft forever preferred_lft forever

inet6 fe80::21f:29ff:fe60:1909/64 scope link

valid_lft forever preferred_lft forever

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:1f:29:60:19:08 brd ff:ff:ff:ff:ff:ff

inet6 fe80::21f:29ff:fe60:1908/64 scope link

valid_lft forever preferred_lft forever

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:1f:29:60:19:0b brd ff:ff:ff:ff:ff:ff

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:1f:29:60:19:0a brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 84:2b:2b:0e:99:58 brd ff:ff:ff:ff:ff:ff

inet 10.10.10.11/24 brd 10.10.10.255 scope global vmbr6

valid_lft forever preferred_lft forever

inet6 fe80::862b:2bff:fe0e:9958/64 scope link

valid_lft forever preferred_lft forever

===================================
ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:54 brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:56 brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:58 brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:5a brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:09 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:08 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:0b brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:0a brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:54 brd ff:ff:ff:ff:ff:ff

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:56 brd ff:ff:ff:ff:ff:ff

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:09 brd ff:ff:ff:ff:ff:ff

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:08 brd ff:ff:ff:ff:ff:ff

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:0b brd ff:ff:ff:ff:ff:ff

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:1f:29:60:19:0a brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 84:2b:2b:0e:99:58 brd ff:ff:ff:ff:ff:ff

============================

ip route

default via 192.168.0.250 dev vmbr0 onlink

10.10.10.0/24 dev vmbr1 proto kernel scope link src 10.10.10.1

10.10.10.0/24 dev vmbr6 proto kernel scope link src 10.10.10.11

192.168.0.0/24 dev vmbr0 proto kernel scope link src 192.168.0.251

192.168.0.0/24 dev vmbr2 proto kernel scope link src 192.168.0.140

James Richmond · Jan 29, 2019

PVE2:

root@pve2:~#

root@pve2:~# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

link/ether 00:26:b9:42:99:86 brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000

link/ether 00:26:b9:42:99:88 brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP group default qlen 1000

link/ether 00:26:b9:42:99:8a brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 00:26:b9:42:99:8c brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP group default qlen 1000

link/ether 00:26:55:e2:f3:b1 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP group default qlen 1000

link/ether 00:26:55:e2:f3:b0 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN group default qlen 1000

link/ether 00:26:55:e2:f3:b3 brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN group default qlen 1000

link/ether 00:26:55:e2:f3:b2 brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:b9:42:99:86 brd ff:ff:ff:ff:ff:ff

inet 192.168.0.252/24 brd 192.168.0.255 scope global vmbr0

valid_lft forever preferred_lft forever

inet6 fe80::226:b9ff:fe42:9986/64 scope link

valid_lft forever preferred_lft forever

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:b9:42:99:88 brd ff:ff:ff:ff:ff:ff

inet 10.10.10.2/24 brd 10.10.10.255 scope global vmbr1

valid_lft forever preferred_lft forever

inet6 fe80::226:b9ff:fe42:9988/64 scope link

valid_lft forever preferred_lft forever

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:55:e2:f3:b1 brd ff:ff:ff:ff:ff:ff

inet 192.168.0.143/24 brd 192.168.0.255 scope global vmbr2

valid_lft forever preferred_lft forever

inet6 fe80::226:55ff:fee2:f3b1/64 scope link

valid_lft forever preferred_lft forever

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:55:e2:f3:b0 brd ff:ff:ff:ff:ff:ff

inet6 fe80::226:55ff:fee2:f3b0/64 scope link

valid_lft forever preferred_lft forever

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:26:55:e2:f3:b3 brd ff:ff:ff:ff:ff:ff

inet6 fe80::226:55ff:fee2:f3b3/64 scope link

valid_lft forever preferred_lft forever

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:26:55:e2:f3:b2 brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:b9:42:99:8a brd ff:ff:ff:ff:ff:ff

inet 10.10.10.12/24 brd 10.10.10.255 scope global vmbr6

valid_lft forever preferred_lft forever

inet6 fe80::226:b9ff:fe42:998a/64 scope link

valid_lft forever preferred_lft forever

27: tap110i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000

link/ether 8a:2d:0e:66:fd:df brd ff:ff:ff:ff:ff:ff

28: tap110i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000

link/ether d2:b5:79:4c:e6:f2 brd ff:ff:ff:ff:ff:ff

29: tap110i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000

link/ether 46:8d:93:29:90:e8 brd ff:ff:ff:ff:ff:ff

root@pve2:~# ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:86 brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:88 brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:8a brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:8c brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b1 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b0 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b3 brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b2 brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:86 brd ff:ff:ff:ff:ff:ff

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:88 brd ff:ff:ff:ff:ff:ff

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b1 brd ff:ff:ff:ff:ff:ff

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b0 brd ff:ff:ff:ff:ff:ff

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b3 brd ff:ff:ff:ff:ff:ff

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e2:f3:b2 brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:b9:42:99:8a brd ff:ff:ff:ff:ff:ff

27: tap110i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN mode DEFAULT group default qlen 1000

link/ether 8a:2d:0e:66:fd:df brd ff:ff:ff:ff:ff:ff

28: tap110i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN mode DEFAULT group default qlen 1000

link/ether d2:b5:79:4c:e6:f2 brd ff:ff:ff:ff:ff:ff

29: tap110i2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000

link/ether 46:8d:93:29:90:e8 brd ff:ff:ff:ff:ff:ff

root@pve2:~# ip route

default via 192.168.0.250 dev vmbr0 onlink

10.10.10.0/24 dev vmbr1 proto kernel scope link src 10.10.10.2

10.10.10.0/24 dev vmbr6 proto kernel scope link src 10.10.10.12

192.168.0.0/24 dev vmbr0 proto kernel scope link src 192.168.0.252

192.168.0.0/24 dev vmbr2 proto kernel scope link src 192.168.0.143

James Richmond · Jan 29, 2019

PVE3:

root@pve3:~# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7b brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7d brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7f brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

link/ether 78:2b:cb:03:eb:81 brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP group default qlen 1000

link/ether 00:26:55:e4:84:e1 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP group default qlen 1000

link/ether 00:26:55:e4:84:e0 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN group default qlen 1000

link/ether 00:26:55:e4:84:e3 brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN group default qlen 1000

link/ether 00:26:55:e4:84:e2 brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7b brd ff:ff:ff:ff:ff:ff

inet 192.168.0.253/24 brd 192.168.0.255 scope global vmbr0

valid_lft forever preferred_lft forever

inet6 fe80::7a2b:cbff:fe03:eb7b/64 scope link

valid_lft forever preferred_lft forever

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7d brd ff:ff:ff:ff:ff:ff

inet 10.10.10.3/24 brd 10.10.10.255 scope global vmbr1

valid_lft forever preferred_lft forever

inet6 fe80::7a2b:cbff:fe03:eb7d/64 scope link

valid_lft forever preferred_lft forever

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:55:e4:84:e1 brd ff:ff:ff:ff:ff:ff

inet 192.168.0.146/24 brd 192.168.0.255 scope global vmbr2

valid_lft forever preferred_lft forever

inet6 fe80::226:55ff:fee4:84e1/64 scope link

valid_lft forever preferred_lft forever

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 00:26:55:e4:84:e0 brd ff:ff:ff:ff:ff:ff

inet6 fe80::226:55ff:fee4:84e0/64 scope link

valid_lft forever preferred_lft forever

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:26:55:e4:84:e3 brd ff:ff:ff:ff:ff:ff

inet6 fe80::226:55ff:fee4:84e3/64 scope link

valid_lft forever preferred_lft forever

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

link/ether 00:26:55:e4:84:e2 brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

link/ether 78:2b:cb:03:eb:7f brd ff:ff:ff:ff:ff:ff

inet 10.10.10.13/24 brd 10.10.10.255 scope global vmbr6

valid_lft forever preferred_lft forever

inet6 fe80::7a2b:cbff:fe03:eb7f/64 scope link

valid_lft forever preferred_lft forever

20: veth200i0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000

link/ether fe:7a:2e:e0:35:71 brd ff:ff:ff:ff:ff:ff link-netnsid 0

root@pve3:~# ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7b brd ff:ff:ff:ff:ff:ff

3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7d brd ff:ff:ff:ff:ff:ff

4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr6 state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7f brd ff:ff:ff:ff:ff:ff

5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:81 brd ff:ff:ff:ff:ff:ff

6: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e1 brd ff:ff:ff:ff:ff:ff

7: enp8s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e0 brd ff:ff:ff:ff:ff:ff

8: enp9s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr4 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e3 brd ff:ff:ff:ff:ff:ff

9: enp9s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast master vmbr5 state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e2 brd ff:ff:ff:ff:ff:ff

10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7b brd ff:ff:ff:ff:ff:ff

11: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7d brd ff:ff:ff:ff:ff:ff

12: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e1 brd ff:ff:ff:ff:ff:ff

13: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e0 brd ff:ff:ff:ff:ff:ff

14: vmbr4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e3 brd ff:ff:ff:ff:ff:ff

15: vmbr5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000

link/ether 00:26:55:e4:84:e2 brd ff:ff:ff:ff:ff:ff

16: vmbr6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

link/ether 78:2b:cb:03:eb:7f brd ff:ff:ff:ff:ff:ff

20: veth200i0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000

link/ether fe:7a:2e:e0:35:71 brd ff:ff:ff:ff:ff:ff link-netnsid 0

root@pve3:~# ip route

default via 192.168.0.250 dev vmbr0 onlink

10.10.10.0/24 dev vmbr1 proto kernel scope link src 10.10.10.3

10.10.10.0/24 dev vmbr6 proto kernel scope link src 10.10.10.13

192.168.0.0/24 dev vmbr0 proto kernel scope link src 192.168.0.253

192.168.0.0/24 dev vmbr2 proto kernel scope link src 192.168.0.146

root@pve3:~#

gotuser32 · Jan 29, 2019

see if your GUI matches your command line output

James Richmond · Jan 29, 2019

gotuser32 said:
see if your GUI matches your command line output

It does.

Took a screenshot but it wont upload.

James Richmond · Jan 29, 2019

There we go.

gotuser32 · Jan 29, 2019

on mine the gui said one thing and the command line said some of the gui info but not the check boxes.
Sorry I am not much help I am very new- not a networking guy by trade now if you got a laser I can help you out LOL

James Richmond · Feb 1, 2019

Stoiko Ivanov said:
Please post the VM config.
Also what do you mean by "using the Intel NIC" - do you have a vmbr for each intel-NIC or do you try to do PCI-passthrough?

* the output of the network-configuration on the node `ip addr`, `ip link`, `ip route` would be helpful, as well as the output of the network configuration from inside the guest `ifconfig`, `route -n` for FreeBSD/pfsense

Do you need any other information Stoiko?

Stoiko Ivanov · Feb 5, 2019

Please use `code` tags for config-files - otherwise it's really hard to get a decent overview.

While it may be unrelated to the current issue - you're using the same network-prefix (192.168.0.0/24) for two interfaces on each server - this does not work (without quite a bit of manual configuration) - please use one network per interface.

Can you ping the PVE-node from the guest (on the interface - 192.168.0.143) - can you ping other PVE-nodes on that interface (this is only reliable after you used each network only once

James Richmond · Feb 6, 2019

Stoiko Ivanov said:
Please use `code` tags for config-files - otherwise it's really hard to get a decent overview.

While it may be unrelated to the current issue - you're using the same network-prefix (192.168.0.0/24) for two interfaces on each server - this does not work (without quite a bit of manual configuration) - please use one network per interface.

Can you ping the PVE-node from the guest (on the interface - 192.168.0.143) - can you ping other PVE-nodes on that interface (this is only reliable after you used each network only once

I will remove the 143,140, 146 from the server. I added them just to see if it would resolve the issue. With that said I could get to the Proxmox with those IP's even though I was not using them. What about CoroSync and Ceph? I have them on a isolated switch from the rest of the network... but both are on the same switch. I don't seem to have any issues with the cluster or any VM that does not use that 4 port NIC. And when I do use that NIC the VM seems fine besides you can't reach it unless you use console with proxmox.

James Richmond · Feb 6, 2019

Code:

2019-02-05 12:27:57.879006 mon.pve1 mon.0 10.10.10.1:6789/0 192680 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.22317s > max 0.05s
2019-02-05 12:27:57.879081 mon.pve1 mon.0 10.10.10.1:6789/0 192681 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.225808s > max 0.05s
2019-02-05 12:27:58.601418 mon.pve1 mon.0 10.10.10.1:6789/0 192682 : cluster [WRN] Health check failed: clock skew detected on mon.pve2, mon.pve3 (MON_CLOCK_SKEW)
2019-02-05 12:28:27.880363 mon.pve1 mon.0 10.10.10.1:6789/0 192688 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.193214s > max 0.05s
2019-02-05 12:28:27.880445 mon.pve1 mon.0 10.10.10.1:6789/0 192689 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.195905s > max 0.05s
2019-02-05 12:29:27.881671 mon.pve1 mon.0 10.10.10.1:6789/0 192703 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.189656s > max 0.05s
2019-02-05 12:29:27.881749 mon.pve1 mon.0 10.10.10.1:6789/0 192704 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.192391s > max 0.05s
2019-02-05 12:30:57.882582 mon.pve1 mon.0 10.10.10.1:6789/0 192730 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.181593s > max 0.05s
2019-02-05 12:30:57.882830 mon.pve1 mon.0 10.10.10.1:6789/0 192731 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.183978s > max 0.05s
2019-02-05 12:32:57.884177 mon.pve1 mon.0 10.10.10.1:6789/0 192751 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.185961s > max 0.05s
2019-02-05 12:32:57.884256 mon.pve1 mon.0 10.10.10.1:6789/0 192752 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.182971s > max 0.05s
2019-02-05 12:35:27.885362 mon.pve1 mon.0 10.10.10.1:6789/0 192778 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.174508s > max 0.05s
2019-02-05 12:35:27.885449 mon.pve1 mon.0 10.10.10.1:6789/0 192779 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.17714s > max 0.05s
2019-02-05 12:38:27.886630 mon.pve1 mon.0 10.10.10.1:6789/0 192821 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.18313s > max 0.05s
2019-02-05 12:38:27.886705 mon.pve1 mon.0 10.10.10.1:6789/0 192822 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.180189s > max 0.05s
2019-02-05 12:41:57.887944 mon.pve1 mon.0 10.10.10.1:6789/0 192863 : cluster [WRN] mon.1 10.10.10.2:6789/0 clock skew 0.168764s > max 0.05s
2019-02-05 12:41:57.888040 mon.pve1 mon.0 10.10.10.1:6789/0 192864 : cluster [WRN] mon.2 10.10.10.3:6789/0 clock skew 0.171508s > max 0.05s
2019-02-05 12:45:58.701124 mon.pve1 mon.0 10.10.10.1:6789/0 192914 : cluster [INF] Health check cleared: MON_CLOCK_SKEW (was: clock skew detected on mon.pve2, mon.pve3)
2019-02-05 12:45:58.701215 mon.pve1 mon.0 10.10.10.1:6789/0 192915 : cluster [INF] Cluster is now healthy

I do see this sometimes but it seems to fix itself. Could this be causing issues?

Stoiko Ivanov · Feb 6, 2019

The clockskew is an issue that needs to be treated (by using the same NTP-server for all your clusternodes, and by using an NTP-Server that is close to them and reliable) - but should be unrelated to the inability of a guest to ping the outside world.

try checking where the packet does not get forwarded after migration - by using tcpdump
`tcpdump -envi vmbr0 icmp1` - to see all packets on interface vmbr0 which are icmp - check the manpage of tcpdump for further infos.

Hope this helps,

Search

Search

NIC card/VM Stop working

James Richmond

Active Member

James Richmond

Active Member

Stoiko Ivanov

Proxmox Staff Member

James Richmond

Active Member

James Richmond

Active Member

James Richmond

Active Member

James Richmond

Active Member

gotuser32

New Member

James Richmond

Active Member

James Richmond

Active Member

Attachments

gotuser32

New Member

James Richmond

Active Member

Stoiko Ivanov

Proxmox Staff Member

James Richmond

Active Member

James Richmond

Active Member

Stoiko Ivanov

Proxmox Staff Member