Validation for OVS network config

[eds89]

Hi,

I'm not the best Linux admin, and this is my first time using Proxmox, and am after some assistance validating my config for my desired network setup.

• I have a TrueNAS box, and two Proxmox boxes, all connected to the same switch.
• Each box has a dual port 10Gb Mellanox card in, and I'm aiming to use one port for a storage network, and one for a LAN network
• For the Proxmox boxes, I'd like the management interface and VMs to be connected to the LAN 10Gb interface
• Vlan ID 1 will be the default untagged VLAN on the switch ports and is what the management interfaces will sit on. Storage I will probably use VLAN 3 but haven't segregated this out yet
• I also have two onboard 1Gb NICs that I'd like to connect directly to the same port on the other Proxmox node, with some VM NICs attached to these for a private VM sync network (some additional VLANs not used elsewhere on the network)

Below is my current config:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr2
#Onboard LAN 1

auto eno2
iface eno2 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr2
#Onboard LAN 2

iface enp65s0 inet manual
        mtu 9000
#ConnectX Port 1

auto enp65s0d1
iface enp65s0d1 inet manual
        ovs_type OVSPort
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options vlan_mode=native-untagged tag=1
#ConnectX Port 2

auto mgmt
iface mgmt inet static
        address 10.0.0.38/24
        gateway 10.0.0.1
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_mtu 9000
        ovs_options tag=1
#Management Interface

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports enp65s0d1 mgmt
        ovs_mtu 9000
#LAN Network

auto vmbr1
iface vmbr1 inet static
        address 172.16.3.2/24
        bridge-ports enp65s0
        bridge-stp off
        bridge-fd 0
        mtu 9000
#Storage Network

auto vmbr2
iface vmbr2 inet manual
        ovs_type OVSBridge
        ovs_ports eno1 eno2
#Private Network

source /etc/network/interfaces.d/*

Image attached is a rough layout of what I am aiming for.

Is my config looking ok for this setup?
I have put the storage interface onto a bridge 1, just in case I want to add an extra adapter in the future, and assigned the 172 storage address to the bridge.
I have created an OVS bridge 0 and added the LAN interface, set vlan_mode to native-untagged with tag 1, along with an OVSIntPort with a VLAN ID of 1. Proxmox management IP on 10.0 network added to this.
Finally, I created bridge 2 with both onboard interfaces attached. No IP assigned

So far, the connectivity to the TrueNAS box on 172 address seems to work, and I have management capability for the Proxmox nodes. It's just that as there appears to be no per-interface network monitoring, I cannot confirm what traffic is using what interface, without physically disconnecting something.

Just hoping someone can confirm if this is all ok, and once I move the storage onto a seprate VLAN and tag the interfaces, it should continue working as expected.
Or, tell me I've missed something so I can go back and review.

Any guidance appreciated.

Many thanks
Eds

 

[Taube]

Hi eds,

can you explain your network concept, e.g. why do you configure a 10gb nic for management and a 1gb nic for network traffic?

greetings taube

 

[eds89]

Hi eds,

can you explain your network concept, e.g. why do you configure a 10gb nic for management and a 1gb nic for network traffic?

greetings taube

Hi Taube,

I am using one port of a 10Gb NIC for storage, and the other port of the 10Gb NIC for VM/Management traffic.
The onboard 1Gb NICs are purely for a private VM sync network between the two Proxmox nodes.

Hope that helps.

Eds