Two Network card

[rfaria]

Hi, I've two network cards in a system, and configured interfaces file like:

auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet dhcp

The cards are on diferents subnets.

When I disconnect one of the two Ethernet cable, the two networks are lost, but if the other cable is disconnected, only the latter fails.

How can i fixit?

 

[wolfgang]

Hi,

i think the problem is your default route.

 

[fleon]

i work with the OP, and the default route 0.0.0.0 goes through eth0. If you unplug the cable from eth0 then you don't get ping replies from eth1. I posted this to another forum and they told me this is expected and the only way would be to configure channel bonding.
Is this true? We even changed both ip's to the same subnet our main computers are in, and the same behavior happens.

How would this be done if proxmox by default uses bridged adapters?

 

[wolfgang]

What is OP?

I'm a bit confuse!
Do you what to make a bond or two independent nics?

 

[fleon]

original poster. We want two independent nics and each one with its own ip, just as windows does by default. if we unplug eth0, we want to continue to be able to ping to the computer through eth1 transparently. we don't want failover.
the main problem seems to be that windows creates a default gateway with each interface, but linux only creates one per machine, not per interface, therefore eth1 depends on eth0 being present

 

[wosp]

That's correct. You can consider to write a simple shell script that monitors eth0, if it's down (no reply on ping i.e.) you let the script delete the default gateway and adds the gateway again using the other interface (eth1). Should be something like:

# ip route del default
# ip route add default via eth1

However, I wonder why you want this. Why not just a bond on eth0 and eth1 and a bridge for the VM's to the bonded interface?

 

[fleon]

If you have two network cards in windows, you can remove the first one and continue to ping the second one. This can't be done with Linux by default. We want the same behavior.

Our branches in our company connect to each other through Frame Relay, Each branch has a 192.168.x.0/24 network. But sometimes the network goes down.

However, each branch also has ADSL internet connection, so we can use a VPN client that allows each branch's computers to continue connection to our network on the main branch.

So basically we want to be able to reach Proxmox through 192.168.x.0/24 (static ip) or through the other network card's ip (dhcp by our ISP)

I found a iproute2 tutorial that i implemented and it seems to have solved the problem.

I edited /etc/iproute2/rt_tables and added: "1 my_route" at the end

Then added the default routes:
ip route add 192.168.x.0/24 dev eth1 src 192.168.x.y table my_route
ip route add default via 192.168.x.1 dev eth1 table my_route
ip rule add from 192.168.x.y/32 table my_route
ip rule add to 192.168.x.y/32 table my_route

Now if i unplug eth0, i can still ping the machine through eth1. Is there a better solution for this?

I tried to setup bonding (i tested with active-backup), but even after the system detected that eth0 went down, i couldn't reach the machine, maybe because of the same issue (routing). However, i don't understand bonding too much, and neither of the 6 bonding modes seemed to apply to our case

That's correct. You can consider to write a simple shell script that monitors eth0, if it's down (no reply on ping i.e.) you let the script delete the default gateway and adds the gateway again using the other interface (eth1). Should be something like:

# ip route del default
# ip route add default via eth1

However, I wonder why you want this. Why not just a bond on eth0 and eth1 and a bridge for the VM's to the bonded interface?

 

[wosp]

Normally with bonding you have 2 (or more) links that can handle the same traffic (i.e. have access to the same VLANs). This way you can eliminate any SPOF in your network and you don't have your users to connect to another IP in case a link fails. For example two switches with both 1 link to the server. The link to switch 1 is primary and the link to switch 2 is secondary (active-backup mode). If switch 1, the cable between switch 1 and server, or the NIC in the server connected to switch 1 fails, the second link will be enabled automatically. In this case you don't set IP's on the interface (eth0/eth1), but on the bond interface or on a bridge that's connected to a bond interface. Let's say your server IP's are 192.168.100.100 and 192.168.101.100, your server is able to connect to the network with these IP's on both eth0 and eth1. Example config for this:

iface eth0 inet manual
  bond_master bond0
  bond_primary eth0
  bond_mode active-backup

iface eth1 inet manual
  bond_master bond0
  bond_primary eth0
  bond_mode active-backup

auto bond0
iface bond0 inet manual
  slaves eth0 eth1
  bond_primary eth0
  bond_mode active-backup
  bond_miimon 100
  bond_updelay 60000

auto bond0.100
iface bond0.100 inet manual
  vlan-raw-device bond0

auto bond0.101
iface bond0.101 inet manual
  vlan-raw-device bond0

auto vmbr0100
iface vmbr0100 inet static
  address 192.168.100.100
  netmask 255.255.255.0
  network 192.168.100.0
  broadcast 192.168.100.255
  gateway 192.168.100.1
  bridge_ports bond0.100
  bridge_stp off
  bridge_fd 0

auto vmbr0101
iface vmbr0101 inet static
  address 192.168.101.100
  netmask 255.255.255.0
  network 192.168.101.0
  broadcast 192.168.101.255
  bridge_ports bond0.101
  bridge_stp off
  bridge_fd 0

This config requires you have 2 VLANs (in this example VLAN 100 for 192.168.100.0/24 and VLAN 101 for 192.168.101.0/24) that both are tagged on both links. If then one link (or switch, or NIC) will fail, both IP's (192.168.100.100 and 192.168.101.100) will still be reachable/working on the local network. As long as the gateway on 192.168.100.1 stay online/reachable, internet traffic will also work over both links. If your switches are the gateway, you can make the gateway redundant with technics like VRRP.

Let's say you also have 192.168.102.0/24 (not directly connected to the PVE node) that needs to be reached over 192.168.101.0/24 and the gateway/router that know how to handle this traffic is on 192.168.101.1, you can simply add a route on the bridge, for example:

auto vmbr0101
iface vmbr0101 inet static
  address 192.168.101.100
  netmask 255.255.255.0
  network 192.168.101.0
  broadcast 192.168.101.255
  bridge_ports bond0.101
  bridge_stp off
  bridge_fd 0
  post-up route add -net 192.168.102.0 netmask 255.255.255.0 gw 192.168.101.1

 

[fleon]

I tried your proposed configuration on a debian vm with 2 network cards, with the only change that i'm trying to get them setup by dhcp.
In the case of stock debian, i had to install bridge-utils and created both bridges with brcrl addbr vmbr0100 and vmbr0101.

From your example i changed static to dhcp, and removed the address, network, broadcast and gateway parameters.

The result is that i don't get addresses anymore, because debian tries to look for a dhcp server and cannot find one.

DHCPDISCOVER on vmbr0101 to 255.255.255.255 port 67 interval 12
No DHCPOFFERS received

Normally with bonding you have 2 (or more) links that can handle the same traffic (i.e. have access to the same VLANs). This way you can eliminate any SPOF in your network and you don't have your users to connect to another IP in case a link fails. For example two switches with both 1 link to the server. The link to switch 1 is primary and the link to switch 2 is secondary (active-backup mode). If switch 1, the cable between switch 1 and server, or the NIC in the server connected to switch 1 fails, the second link will be enabled automatically. In this case you don't set IP's on the interface (eth0/eth1), but on the bond interface or on a bridge that's connected to a bond interface. Let's say your server IP's are 192.168.100.100 and 192.168.101.100, your server is able to connect to the network with these IP's on both eth0 and eth1. Example config for this:

iface eth0 inet manual
  bond_master bond0
  bond_primary eth0
  bond_mode active-backup

iface eth1 inet manual
  bond_master bond0
  bond_primary eth0
  bond_mode active-backup

auto bond0
iface bond0 inet manual
  slaves eth0 eth1
  bond_primary eth0
  bond_mode active-backup
  bond_miimon 100
  bond_updelay 60000

auto bond0.100
iface bond0.100 inet manual
  vlan-raw-device bond0

auto bond0.101
iface bond0.101 inet manual
  vlan-raw-device bond0

auto vmbr0100
iface vmbr0100 inet static
  address 192.168.100.100
  netmask 255.255.255.0
  network 192.168.100.0
  broadcast 192.168.100.255
  gateway 192.168.100.1
  bridge_ports bond0.100
  bridge_stp off
  bridge_fd 0

auto vmbr0101
iface vmbr0101 inet static
  address 192.168.101.100
  netmask 255.255.255.0
  network 192.168.101.0
  broadcast 192.168.101.255
  bridge_ports bond0.101
  bridge_stp off
  bridge_fd 0

This config requires you have 2 VLANs (in this example VLAN 100 for 192.168.100.0/24 and VLAN 101 for 192.168.101.0/24) that both are tagged on both links. If then one link (or switch, or NIC) will fail, both IP's (192.168.100.100 and 192.168.101.100) will still be reachable/working on the local network. As long as the gateway on 192.168.100.1 stay online/reachable, internet traffic will also work over both links. If your switches are the gateway, you can make the gateway redundant with technics like VRRP.

Let's say you also have 192.168.102.0/24 (not directly connected to the PVE node) that needs to be reached over 192.168.101.0/24 and the gateway/router that know how to handle this traffic is on 192.168.101.1, you can simply add a route on the bridge, for example:

auto vmbr0101
iface vmbr0101 inet static
  address 192.168.101.100
  netmask 255.255.255.0
  network 192.168.101.0
  broadcast 192.168.101.255
  bridge_ports bond0.101
  bridge_stp off
  bridge_fd 0
  post-up route add -net 192.168.102.0 netmask 255.255.255.0 gw 192.168.101.1

 

[wosp]

The config example I provided is for use on the PVE node. After that you assign the vmbr interfaces to the VM's.

 

[fleon]

The solution was to use a second routing table, using iproute2, leaving vmbr0 as dhcp (wan) and vmbr1 as static (lan)

echo "1 my_route" >> /etc/iproute2/rt_tables
ip route add 192.168.x.0/24 dev vmbr1 src 192.168.x.y table my_route
ip route add default via 192.168.x.1 dev vmbr1 table my_route
ip rule add from 192.168.x.y/32 table my_route
ip rule add to 192.168.x.y/32 table my_route

With this configuration, we can unplug the cable from vmbr0 and still vmbr1 will respond. Granted, this isn't failover per se, but it's what we wanted.
I still don't understand why this is needed. Windows does this by default.

 

[1nerdyguy]

Mostly because this isn't windows.