[SOLVED] Setting up multiple nic Network and routing on Proxmox

[Ultraformer]

Just install my Proxmox server. My server has a dual Nic. Each nic port is connected to a VLAN switch port that serves two separate VLANs (port 2 for VLAN 10, port 3 for VLAN 100). The topology is below:

Here is what is on the console

on the GUI

and in the /etc/network/interfaces


I am building VMs for both VLANs. The router will be doing all the routing between the VLANS. Questions I have:

1, Do I need to manually enable eno2? (when there is a link light on the switch.) Do I need to assign an IP address to eno2?
2, Do I need to add a static route for each VLAN on the host?
3. When I try to create a VMBR1 for 192.168.100.x network, the GUL failed and said the GW already exists. How can I make that work?
- can I just copy the VMBR0 config and modify it for the 192.168.100.x network?

Thank you very much for any help. I am new to Proxmox and did not need to do resolve networking when I was using VMware. Just trying to get a sense of what I should be doing.

 

[SCrisler]

You do need to create VMBR1 for 192.168.100.x. Doing that will enable device eno2. Although it is not intuitively obvious, the trick is to leave the gateway field blank on VMBR1. If you don't have the network up down installed, you will have to reboot the node to apply the network changes.

I don't have an in depth understanding, but I think there can only be one gateway per node. I'm not sure about the cross network routing. Anytime I have used a VPN, the traffic has remained contained to the VPN network on the switch.

Proxmox does have VPN support as part of it's networking but since I have never done it I can only point out that it exists and refer you to the documentation.

 

[bobmc]

Your host management IP 192.168.10.10 is on VLAN 10 but as it stands, you have no way of tagging the management traffic for the correct VLAN unless you make the switch port it is connected to untagged (in Procurve parlance) or access (in Cisco parlance) for VLAN 10.

You can create VLAN ports in Proxmox if required

auto lo
iface lo inet loopback

iface eno1 inet manual


auto vmbr0.10
iface vmbr0.10 inet static
        address  192.168.10.10
        netmask  255.255.255.0
        gateway  192.168.10.1  

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes

This would mean anything assigned to use vmbr0.10 would be tagged for VLAN 10

I would recommend you reserve eno1 for management traffic (possibly not on VLAN 10 or VLAN 100 but it depends on your network config goals), and use eno2/vmbr1 for all VM traffic. Make that bridge 'VLAN aware' and then tag the desired VLAN in the network page of the VM or Container
That's a more flexible approach.

https://pve.proxmox.com/wiki/Network_Configuration

 

[AllanM]

Worth pointing out that you don't need to use up both NIC ports for these internal network connections... proxmox can do vlans on an individual interface very easily...

Personally, to make things easy to keep track of, I always create a vmbr1 and use it as the management network for the environment on vlan1, (I move the proxmox node management net to vmbr1) leaving vmbr0 blank with no config and no interface assignment. I then create additional vmbr's each named with the vlan number they will be used for, and I also like to use addressing schemes that tell me something about the vlan like you've done there...

Example:
vmbr0 - nothing
vmbr1: eno1 10.1.1.61/24 10.1.1.1
vmbr10: eno1.10 (tagged vlan 10)
vmbr100: eno1.100 (tagged vlan 100)

If you need dedicated bandwidth, then certainly, you can break it out to more interfaces.....

 

[Ultraformer]

Worth pointing out that you don't need to use up both NIC ports for these internal network connections... proxmox can do vlans on an individual interface very easily...

Personally, to make things easy to keep track of, I always create a vmbr1 and use it as the management network for the environment on vlan1, (I move the proxmox node management net to vmbr1) leaving vmbr0 blank with no config and no interface assignment. I then create additional vmbr's each named with the vlan number they will be used for, and I also like to use addressing schemes that tell me something about the vlan like you've done there...

Example:
vmbr0 - nothing
vmbr1: eno1 10.1.1.61/24 10.1.1.1
vmbr10: eno1.10 (tagged vlan 10)
vmbr100: eno1.100 (tagged vlan 100)

If you need dedicated bandwidth, then certainly, you can break it out to more interfaces.....

Thank you. I guess I will properly do that. Bandwidth should not be an issue as it is a dual 10G ethernet connection. I guess I will move the management port to the 1G port for vlan1 and use a single 10G port for both VLAN. I should not expect to use much bandwidth for management other than copying ISOs.

 

[Ultraformer]

Your host management IP 192.168.10.10 is on VLAN 10 but as it stands, you have no way of tagging the management traffic for the correct VLAN unless you make the switch port it is connected to untagged (in Procurve parlance) or access (in Cisco parlance) for VLAN 10.

You can create VLAN ports in Proxmox if required

auto lo
iface lo inet loopback

iface eno1 inet manual


auto vmbr0.10
iface vmbr0.10 inet static
        address  192.168.10.10
        netmask  255.255.255.0
        gateway  192.168.10.1 

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes

This would mean anything assigned to use vmbr0.10 would be tagged for VLAN 10

I would recommend you reserve eno1 for management traffic (possibly not on VLAN 10 or VLAN 100 but it depends on your network config goals), and use eno2/vmbr1 for all VM traffic. Make that bridge 'VLAN aware' and then tag the desired VLAN in the network page of the VM or Container
That's a more flexible approach.

https://pve.proxmox.com/wiki/Network_Configuration

Thank you for the suggestion. I will be setting up VLAN and move the management interface to VLAN1 (as with most of my management portals for my networking gear).

 

[AllanM]

Thank you. I guess I will properly do that. Bandwidth should not be an issue as it is a dual 10G ethernet connection. I guess I will move the management port to the 1G port for vlan1 and use a single 10G port for both VLAN. I should not expect to use much bandwidth for management other than copying ISOs.

You can carry management vlan1 on the same 10G port with the others if you want. Tagged or untagged, however you want to set it up. Just match it on that port to the switch.

It does help with uploading big ISOs

 

[Ultraformer]

You can carry management vlan1 on the same 10G port with the others if you want. Tagged or untagged, however you want to set it up. Just match it on that port to the switch.

It does help with uploading big ISOs

Yeah. I have it on 10G now. Once I upgrade my NAS, I will need to free up one of the 10G ports. The 10G does make uploading ISO a breeze. I think I will physically separate the connections. This is my only dual home device and I use a firewall to separate the traffic between VLANs.

What I end up doing:

VMBR0: management interface from one nic port (VLAN1). Created during setup
VMBR1: make it VLAN aware with no IP information and connect the second nic port to a trunk port on the switch (I did limit the Vlan ID to VLAN10 and 100 on the /etc/network/interfaces file in Proxmox)

Tested and it works. No need to have sub-interfaces create. Just need to make sure the VLAN is tagged when creating the VM.

Properly going to have some more testing and learning the difference between Proxmox and VMWare.

Thank you very much for your help and information.

 

[AllanM]

I've always found it easier to deal with all the vlan definition right down at the hypervisor node config, rather than up in the VM configs. I think it's more versatile as it allows you to connect a VM to multiple networks by adding more virtual NICs, you know, like if you want to virtualize a router or have a VM that provides services on 2 networks or create a virtualized span/sensor setup for network monitoring.

I always create a bridge for every vlan in the environment on every node in a proxmox cluster. Sometimes I'll even throw in additional "future use" bridges just so they are ready to go when the need arises.

 

[Ultraformer]

I've always found it easier to deal with all the vlan definition right down at the hypervisor node config, rather than up in the VM configs. I think it's more versatile as it allows you to connect a VM to multiple networks by adding more virtual NICs, you know, like if you want to virtualize a router or have a VM that provides services on 2 networks or create a virtualized span/sensor setup for network monitoring.

I always create a bridge for every vlan in the environment on every node in a proxmox cluster. Sometimes I'll even throw in additional "future use" bridges just so they are ready to go when the need arises.

Very good point. Haven't thought about future proof of the settings.

So here is what I change to :

VMBR0: management (unchanged) with bridge port to eno1
VMBR1: VLAN Aware with bridge port to eno2
VMBR10: VLAN aware with bridge port to eno2.10
VMBR100: VLAN aware with bridge port to eno2.100

Tested and it works. Thank you again for your help

 

[AllanM]

No need for the "vlan aware" flag

 

[compuls1v3]

Thanks to everyone for explaining how to setup vlans! I was also struggling to wrap my head around it and the knowledge on here was very helpful!

 

[compuls1v3]

No need for the "vlan aware" flag

How does the bridge know about vlans without that flag then? Just curious.