dual nic issue

[booradlly]

I have two nics: a 1g mb nic, and a 10g mellanox nic with two ports.

I want to use the 10g mellanox card for everything but management, and if the mellanox card dies, I want to still be able to remote into the box. Currently, I don't see how thats possible.

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp5s0 inet manual

iface enp6s0 inet manual

auto enp4s0
iface enp4s0 inet manual

auto enp4s0d1
iface enp4s0d1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp4s0 enp4s0d1
        bond-miimon 100
        bond-mode balance-alb
#mellanox

auto vmbr0
iface vmbr0 inet static
        address x.x.x.30/24
        gateway x.x.x.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

iface wlp7s0 inet manual

auto vmbr1
iface vmbr1 inet static
        address x.x.x.31/24
        bridge-ports enp5s0
        bridge-stp off
        bridge-fd 0
#VM traffic

/etc/network/interfaces (END)

More detail:
During install a bridge was created for the 1g mb nic, with a default gateway.
I then attempted to get the 10g mellanox nic up and running, by creating a bridge for that.
It appeared to work, I was able to ping the new IP, and open the web gui, but then I unplugged the 1g mb nic, and both IPs stopped working, and my server was unreachable.
I went through numerous iterations and saw that the 10g mellanox nic was just not being used.
After more iterations, I realized the first bridge was taking precedence for some reason.
I swapped the bridge ports, and now the 10g mellanox is being used. iperf shows 10gbits /sec.
However, if I unplug the mellanox now, the 1g mb nic is useless.

Normally, if I install two nics, and give them two IPs, they are independent.

With Proxmox, its like it only allows one nic.

I feel like Im missing something major.

I want to use the 10g mellanox card for everything but management, and if the mellanox card dies, I want to still be able to remote into the box. Currently, I don't see how thats possible.

At this point I am assuming its not possible to have two nics working independently from eachother in proxmox. As this is a pretty standard redundant configuration in production, I still feel I am missing something. Proxmox is used in production environments, so I am at a loss. Is this something that needs proxmox installed on debian, instead of the proxmox direct install like I did ?

 

[Dunuin]

It appeared to work, I was able to ping the new IP, and open the web gui, but then I unplugged the 1g mb nic, and both IPs stopped working, and my server was unreachable.

If you remove a PCIe device (for example by removing a physical card or disableing onboard devices in BIOS) all NICs will change the name and your previous network config won't work any longer and you need to edit the "/etc/network/interfaces" so the config matches the new NIC names.

Normally, if I install two nics, and give them two IPs, they are independent.

Also make sure you don't assign two NICs IPs in the same subnet or that can cause problems because it isn't clear which NIC to use when talking to another host in that subnet.

I want to use the 10g mellanox card for everything but management, and if the mellanox card dies, I want to still be able to remote into the box. Currently, I don't see how thats possible.

In that case I would only set the IP and gateway to the bridge with the Gbit NIC. You don't need a IP on the 10Gbit bridge if it is just for the VM.
And if you want to make sure that the network config is still working if a NIC fails you should setup some udev rules so that for example eth0 is always mapped to the same physical NIC. See here: https://wiki.debian.org/NetworkInterfaceNames#THE_.22PERSISTENT_NAMES.22_SCHEME

 

[booradlly]

[...] if you want to make sure that the network config is still working if a NIC fails you should setup some udev rules so that for example eth0 is always mapped to the same physical NIC. See here: https://wiki.debian.org/NetworkInterfaceNames#THE_.22PERSISTENT_NAMES.22_SCHEME

Wow this is insane, the idea that a nic would change its name on me was never even on my radar. I'm new to Debian, but not RHEL, do you know if this is a Debian thing, or have I been missing something major about Linux in general for a while now ? What about FreeBSD ?
I wasn't actually talking about removing a nic, was talking about cutting a cord. But, that doesnt really matter if the card dies, and also takes out my ability to remote in, so thank you for this.

[...] In that case I would only set the IP and gateway to the bridge with the Gbit NIC. You don't need a IP on the 10Gbit bridge if it is just for the VM.
[...]

In this scenario, wouldn't I be SOL, if the 1g mb nic network cable was pulled ?
The goal is if the 10g mellanox card cord gets pulled, I want to still be able to remote into the box and conversely,
if the 1g mb nic cord gets pulled, I want to still be able to remote into the box.
Basically, I have multiple nics in this server, if multiple cables get cut, I want to be able to remote in regardless, as long as one cable has not been cut.
I would think this is a pretty standard scenario, or am I missing something thats making this a lot more complicated.

[...] Also make sure you don't assign two NICs IPs in the same subnet or that can cause problems because it isn't clear which NIC to use when talking to another host in that subnet.
[...]

I found one post that said Im describing a bond, and that I should bond the 1g and 10g interfaces, and let STP handle using the fastest lane.
I have found other posts that said not to bond 1g and 10g interfaces.

 

[Dunuin]

Wow this is insane, the idea that a nic would change its name on me was never even on my radar. I'm new to Debian, but not RHEL, do you know if this is a Debian thing, or have I been missing something major about Linux in general for a while now ? What about FreeBSD ?
I wasn't actually talking about removing a nic, was talking about cutting a cord. But, that doesnt really matter if the card dies, and also takes out my ability to remote in, so thank you for this.

Thats totally normal. Previously the NICs were just named eth0, eth1, eth2 and so on so that basically the first NIC found will be eth0, the second found eth1, ... . So as soon as you add/remove a NIC the order might change and the old names might refer to other physical NICs.
Later many linuxes switched to "predictable names" where the NIC will be named after how it is connected to the mainboard. Se here: https://major.io/2015/08/21/understanding-systemds-predictable-network-device-names/
But there the names can still switch in case a NIC is failing and isn't recognized by the mainbaord anymore.

In this scenario, wouldn't I be SOL, if the 1g mb nic network cable was pulled ?
The goal is if the 10g mellanox card cord gets pulled, I want to still be able to remote into the box and conversely,
if the 1g mb nic cord gets pulled, I want to still be able to remote into the box.
Basically, I have multiple nics in this server, if multiple cables get cut, I want to be able to remote in regardless, as long as one cable has not been cut.
I would think this is a pretty standard scenario, or am I missing something thats making this a lot more complicated.

I found one post that said Im describing a bond, and that I should bond the 1g and 10g interfaces, and let STP handle using the fastest lane.
I have found other posts that said not to bond 1g and 10g interfaces.

Jup, what you need is a bond with failover capabilities but bonding 1Gbit and 10Gbit can be problematic, especially when using a bond type with load balancing and not just a "active-backup" type bond where only one NIC is active at a time.