Multiple Network Adaptors / Multiple Gateways

[bgeorge0176]

I am setting up a PVE for the first time and am trying to understand how to properly set up the network. I have 6 network adaptors, 2 10GB SFP+ and 4 1GB ethernet, that I want to bond into 2 specific networks:

1. Management network (10.254.x.x) - for just that, management access
2. Server network (10.200.x.x) - for communication between servers in cluster and shared storage array
3. Presentation network (10.250.x.x) - for client access to servers and virtual desktop environment

My understanding is that, upon installation, the gateway set for the primary interface (in my case, management), is the only gateway that can exist. I was able to edit /etc/network/interfaces and apply gateways to the other two networks, but I guess my question is, what is the best practice there? Each network is configured with its own unique gateway at the Layer 3 level, so I would assume they each need the gateway configured in PVE. If not, what is the proper way or, better yet, best practice to setting up the networks.

 

[louie1961]

If all of your VLANs/networks are on the same router, then only one gateway is required. Here is how I did it, but this is my first Proxmox bond, so I am not 100% sure this is the best way. But it works for me:

auto lo
iface lo inet loopback

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

auto bond0
iface bond0 inet static
bond-slaves enp1s0f0 enp1s0f1
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate fast
bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092

auto vmbr0.10
iface vmbr0.10 inet static
address 10.10.10.2/24
gateway 10.10.10.1

source /etc/network/interfaces.d/*

By doing it this way, my Proxmox web UI is on 10.10.10.2. I can set all of my VMs and/or LXC containers to whatever VLAN I want in the hardware tab for each.

 

[UdoB]

My understanding is that, upon installation, the gateway set for the primary interface (in my case, management), is the only gateway that can exist.

Yes.

I was able to edit /etc/network/interfaces and apply gateways to the other two networks

Note: I am not sure if I understand your construct correctly. Post your /etc/network/interfaces (in code-tags) to avoid misunderstandings on my side.

That does not make sense at all! There can be only one single default gateway. And if the node needs to have an IP address inside those other networks for a specific reason an address for the bridge is sufficient - NO gateway required or involved.

Usually the PVE node does NOT have an IP address in each and every secondary network at all! You need to prepare a vmbr123 (on CLI or in the Gui) with one physical interface. That's it. The VMs can use it, with whatever IP network they choose. Without the host knowing about it (on this IP level)!

Of course you need some kind of router to interconnect separate networks. But that is not the primary task for the PVE host! This may be done by the PVE node, but personally I prefer a "real" router, like OPNsense or just Debian with shorewall.