Differences in networking setup between v4 & v5?

[ejmerkel]

We've been installing Proxmox V4 nodes with redundant Juniper EX4550 switches in a stacked configuration for some time with no issues at all. We generally setup each host as follows:

bond0 (OVS Bond (vmbr1) - eth0 & eth1) - various VLAN's used for VM's public network access
bond1 (Linux Bond - eth2 & eth3 - LACP) - Managment network, clustering, etc
bond2 (Linix Bond - eth4 & eth5 - LACP) - Storage Network

We recently configured some new Proxmox v5 nodes in the same network configuration but I cannot get the either the control (bond1) or storage network (bond2) to come up and be pingable. Our switch technicians swears that the ports are configured the same as the V4 nodes but they still don't work. If I plug my laptop directly into the either bond port on the servers, I can ping the IP addresses.

Is there any difference between v4 & v5 that I might be missing that would cause the links not to come up?

Here are the proxmox packages.

proxmox-ve: 5.2-2 (running kernel: 4.15.17-2-pve)
pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
pve-kernel-4.15: 5.2-2
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-32
libpve-guest-common-perl: 2.0-16
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 0.6-4
openvswitch-switch: 2.7.0-2
proxmox-widget-toolkit: 1.0-18
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-9
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-5
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5

Unfortunately, I only have remote console via IDRAC so can't copy network config easily in this post, but here are links to 2 screen captures of the /etc/network/interfaces.

https://prntscr.com/k5aysn
https://prntscr.com/k5aynv

Note: vmbr0 is not being used in this configuration.

Any ideas on how to trouble shoot this from the proxmox side?

Thanks,
Eric

 

[wolfgang]

Hi,

the ovs version is another one.
Why do you use ovs? You have a simple setup and can easily use Linux bridge model.

 

[ejmerkel]

Hi,

the ovs version is another one.
Why do you use ovs? You have a simple setup and can easily use Linux bridge model.

I have always used OVS in the past for our customer VLAN's but the problem I am having is on the Linux bonds. Do I have the wrong version of OVS installed? Could that be what is causing the the Linux bonds not to work?

Best regards,
Eric

 

[wolfgang]

The ovs package seems to be correct.
What nics do you use? The kernel do you use is known for an Intel igb bug.
This is fixed in pve-kernel-4.15.17-3-pve: 4 .15.17-14

 

[ejmerkel]

The ovs package seems to be correct.
What nics do you use? The kernel do you use is known for an Intel igb bug.
This is fixed in pve-kernel-4.15.17-3-pve: 4 .15.17-14

I am using 1G igb interfaces but does this also affect the 10G interfaces as well? Here is a screen print of my interfaces http://prntscr.com/k5la6i

Best regards,
Eric

 

[ejmerkel]

I've tried rebooting to an older kernel to see if the network interface comes up but they don't. I am seeing the following error message on the bond interfaces. Could this be a clue to the issue?

http://prntscr.com/k5nffs
http://prntscr.com/k5ng5v

 

[ejmerkel]

I've upgraded to 4.15.17-3-pve #1 SMP PVE 4.15.17-4 kernel dated June 27th and rebooted... still no luck on the bonding.

cat /proc/net/bonding/bond2 shows the bond and slaves being up but can't ping so so any ideas how to trouble shoot further on the proxmox side?

Best regards,
Eric

 

[wolfgang]

If you use OVS switch you should use OVS Bonds and not Linux bonds.
You are mixing two technologies what never be tested to work together.

 

[ejmerkel]

Just to follow up on this thread. This problemt turned out to be a Juniper issue. Basically they don't allow you to create aggregated ethernet interfaces with numbers above 511. Our interfaces were created on the switch but never went into affect because of this limitation. That's why all of our interfaces below 511 worked and these did not. You would think the switch would complain but it would happily accept the config and ignore it.

In any case all is working now. Sorry to trouble you.

Eric