network interfaces dual port

[Iceman13]

Hey Proxmox crew,

i wanted to create two vmbr with an Ethernet port each. it somewhat works with the config that i have but i get failed to raise network interface on startup.
here is my etc/network/interfaces file

auto lo
iface lo inet loopback

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.100
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports enp4s0
bridge_stp off
bridge_fd 0

iface enp5s0 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.1.110
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports enp5s0
bridge_stp off
bridge_fd 0

i read somewhere that two gateways is not correct. any recomendations
thanks

 

[gurubert]

I would not recommend to attach two interfaces with different IPs to the same VLAN. The Linux kernel will answer ARP requests for any of them and you get ARP flapping in the network and unreliable connections.

Better create a bonding Interface with the two Ethernet interfaces and then a bridge on top of that with both IP addresses assigned. This way you also do not get confused with two default gateways which is usually not possible.

 

[Iceman13]

What

I would not recommend to attach two interfaces with different IPs to the same VLAN. The Linux kernel will answer ARP requests for any of them and you get ARP flapping in the network and unreliable connections.

Better create a bonding Interface with the two Ethernet interfaces and then a bridge on top of that with both IP addresses assigned. This way you also do not get confused with two default gateways which is usually not possible.

hi, thanks for the reply
what should my etc/network file look like then?
thanks

 

[Dunuin]

i wanted to create two vmbr with an Ethernet port each. it somewhat works with the config that i have but i get failed to raise network interface on startup.
here is my etc/network/interfaces file

Why do you want that in the first place?
I dont see any benefit doing that.
There would be no redundancy/failover because each VM still relies on that single NIC it is attached to. And you dont increase security because there is still no isolation if you want both NICs to be in the same subnet. So any VM on vmbr0 can still access any VM on vmbr1. In such a case a round robin or better LACP layer3+4 bond would be better offering failover and more flexible bandwidth because any two VMs could use 2x 1 Gbit instead of only one VM from vmbr0 + one VM of vmbr1.

And like iceman13 already said, you shouldn't setup two gateways or two IPs from the same subnet range on a single host.

 

[Iceman13]

Why do you want that in the first place?
I dont see any benefit doing that.
There would be no redundancy/failover because each VM still relies on that single NIC it is attached to. And you dont increase security because there is still no isolation if you want both NICs to be in the same subnet. So any VM on vmbr0 can still access any VM on vmbr1. In such a case a round robin or better LACP layer3+4 bond would be better offering failover and more flexible bandwidth because any two VMs could use 2x 1 Gbit instead of only one VM from vmbr0 + one VM of vmbr1.

And like iceman13 already said, you shouldn't setup two gateways or two IPs from the same subnet range on a single host.

I set this up as few years ago when i ddint understand much. It thought i would get better bandwith by seperating more impotant vms or less of them on vmbr0 and then the rest on vmbr1. if i want bonding what should the settings look like? thanks

 

[Dunuin]

Do you got a managedswitch that supports LACP? If not you could try round robin or another of the bonding modes.

You could try something like that:

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

iface bond0 inet manual
   bond-slaves enp4s0 enp5s0
   bond_miimon 100
   bond_mode balance-rr

auto vmbr0
iface vmbr0 inet static
   address 192.168.1.100
   netmask 255.255.255.0
   gateway 192.168.1.1
   bridge_ports bond0
   bridge_stp off
   bridge_fd 0

But make sure to backup your config file (cp /etc/network/interfaces /etc/network/interfaces.backup) and have a keyboard+monitor or WebKVM available if it doesn't work if you lock you out from SSH/WebUI.

 

[Iceman13]

Do you got a managedswitch that supports LACP? If not you could try round robin or another of the bonding modes.

You could try something like that:

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

iface bond0 inet manual
   bond-slaves enp4s0 enp5s0
   bond_miimon 100
   bond_mode balance-rr

auto vmbr0
iface vmbr0 inet static
   address 192.168.1.100
   netmask 255.255.255.0
   gateway 192.168.1.1
   bridge_ports bond0
   bridge_stp off
   bridge_fd 0

But make sure to backup your config file (cp /etc/network/interfaces /etc/network/interfaces.backup) and have a keyboard+monitor or WebKVM available if it doesn't work if you lock you out from SSH/WebUI.

hey thanks for the reply ill try that
also would this be the correct way to just do one port?
auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.100
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports ennp4s0
bridge_stp off
bridge_fd 0

if i were to do the bond. do i have to enable link aggregation on my router? i have an asus ax11000
thanks

 

[Dunuin]

also would this be the correct way to just do one port?
auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.100
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports ennp4s0
bridge_stp off
bridge_fd 0

There is a typo "ennp4s0", but beside that that looks fine.

if i were to do the bond. do i have to enable link aggregation on my router? i have an asus ax11000
thanks

If you got link aggregation in your router I would use LACP. In that case you need to enable it in your router and configure how it should operate. IT should match the PVE network config (like hashing policy=layer3+4).

It could look like this:

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

iface bond0 inet manual
   bond-slaves enp4s0 enp5s0
   bond_miimon 100
   bond-mode 802.3ad
   bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
   address 192.168.1.100
   netmask 255.255.255.0
   gateway 192.168.1.1
   bridge_ports bond0
   bridge_stp off
   bridge_fd 0

 

[Iceman13]

There is a typo "ennp4s0", but beside that that looks fine.

If you got link aggregation in your router I would use LACP. In that case you need to enable it in your router and configure how it should operate. IT should match the PVE network config (like hashing policy=layer3+4).

It could look like this:

auto lo
iface lo inet loopback

iface enp4s0 inet manual

iface enp5s0 inet manual

iface bond0 inet manual
   bond-slaves enp4s0 enp5s0
   bond_miimon 100
   bond-mode 802.3ad
   bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
   address 192.168.1.100
   netmask 255.255.255.0
   gateway 192.168.1.1
   bridge_ports bond0
   bridge_stp off
   bridge_fd 0

i see the typo thank you.
i will try to get the single one working for now.
the lacp i understand but i dont understand some of the codes in that file. like the layer 3 and 4
the rest i understand i think.
would i enable on router first and the proxmox or vice versa?
thanks for all the help

 

[Dunuin]

i see the typo thank you.
i will try to get the single one working for now.
the lacp i understand but i dont understand some of the codes in that file. like the layer 3 and 4
the rest i understand i think.

Then you should learn that. Buying such an expensive router is a waste if you don't use its advanced features because you don't understand them.

would i enable on router first and the proxmox or vice versa?
thanks for all the help

I would do it on the PVE first and then on the router.

 

[Iceman13]

Then you should learn that. Buying such an expensive router is a waste if you don't use its advanced features because you don't understand them.

I would do it on the PVE first and then on the router

Yeah im still learning all of the features.
I tried doing the bond through the command line and it didn’t work.
Could not reach the web ui and got a network interfaces error during boot.
I’ll try to do it again in the web ui and see where I went wrong.
Thanks

 

[Dunuin]

That should be normal. If you tell PVE to use LACP bond and didn't changed it on your router to LACP too, then the bond won't work. Only if both your PVE and router are correctly setup to use LACP your PVE host will be reachable. So don't forget to set it up in the router right after doing it on your PVE.
Also make sure to directly connect your both ethernet cables from your PVE server to your router (and use the excat right ports on your router which you set to operate as a LACP bond) without a switch in between or that bond wont work.

 

[Iceman13]

That should be normal. If you tell PVE to use LACP bond and didn't changed it on your router to LACP too, then the bond won't work. Only if both your PVE and router are correctly setup to use LACP your PVE host will be reachable. So don't forget to set it up in the router right after doing it on your PVE.
Also make sure to directly connect your both ethernet cables from your PVE server to your router (and use the excat right ports on your router which you set to operate as a LACP bond) without a switch in between or that bond wont work.

Both Ethernet cables are attached to the two rj45 ports. I have an asus z10pa d8. From what I understand the two ports next to each other are lan and the third is used for the ikvm module.
I set it up on proxmox and then shutdown the system. I enabled link aggregation on the router under lan and switch control. Ports one and 2 are what the manual says. I then started proxmox up and got an error. So I cli back to original settings to one port and booted back up and it worked. I might have missed something in the cli to get the error. I should be able to try in the web ui to create the bond

 

[Dunuin]

First I would verify that you configured your router right. Did you for example set the router to "layer3+4" hashing too? They both need to work in the same mode.

 

[Iceman13]

First I would verify that you configured your router right. Did you for example set the router to "layer3+4" hashing too? They both need to work in the same mode.

thats what i am trying to figure out now. i cannot find the layer (2+3) or 4 settings anywhere on the router. it just says enable link aggregation yes or no. also in the slaves column in the web ui under bond do i separate them with commas or just a space? enp4s0 enp5so or enp4so, enp5s0

 

[Dunuin]

Should be just a space. Looks like your router doesn't support using different hashing modes. So you can only try to switch between modes on your server until you find one that works with your router. You could for example try "layer2+3" then instead of "layer3+4".

 

[Iceman13]

I’m going to try layer 2+3
When I tried to do it through the web ui it says enp4s0 is in use. Should I try cli again and then reboot or is there another method

 

[Dunuin]

You cant make a bond of a NIC that is used for something else. If you want to make a bond with that NIC you would need to remove that NIC from your bridges first and add bridge that bridge to your bond.

 

[Iceman13]

You cant make a bond of a NIC that is used for something else. If you want to make a bond with that NIC you would need to remove that NIC from your bridges first and add bridge that bridge to your bond.

I understand that part. Will I lose connection to the server if I remove the bridge?
I can shut the vms down and that’s ok but if I remove the bridge will the ip adress in the web io still work to then create the bond?
Thanks

 

[Dunuin]

WebUI and SSH won't work if you remove the NIC from the bridge and your IP is set to that bridge. You should install ifupdown2 and try to remove the NIC from the bridge, create the bond with both NICs and add that bond to the bridge at the same time and then hit the apply button. If that doesn't work you can only edit the /etc/network/interfaces by CLI.