Setting up multiple networks between PVE

[nightingale1981]

Hi !

I have installed a PVE hypervisor on the physical server.It is connected to the enterprise's shared network through the vmbr0 virtual switch created by default.
An IP address associated with vmbr0 has been assigned.My idea was to create a virtual machine inside the PVE and make this VM a virtual router.
After creating the VM, I connected vmbr0 to it and assigned it the same IP address that was previously linked to the PVE management.
Then I created 3 more virtual switches (vmbr1, vmbr2, vmbr3) and added them to the virtual router.
vmbr1 – for communication with other hardware hypervisors (clustering)
vmbr2 – for server VMs
vmbr3 – for virtual workstationsI assigned each of them my own IP address of a separate subnet and enabled address forwarding (IP forward).

Now, traffic from the enterprise network enters the virtual router, from which it is routed via vmbr1 to the PVE management.And the rest (vmbr2 and vmbr3) will be used only for accessing virtual machines.And now the task has appeared to install a second (and in the future a third) PVE, connect PVE1 and PVE2 through one physical server port. Connect them to each other via a switchboard.
And vmbr2 and vmbr3 can also be used on the second PVE.
But here's the problem... PVE does not allow you to pass multiple virtual switches through one physical server port. Only 1 to 1.
I'm not a network technology specialist, so I'm asking for your help.I would like to make sure that all 3 networks are connected to the rest of the PVE hosts. If I combine two PVE hosts into a cluster and the connection between them will be only through vmbr1 (clustering), then will the others (vmbr2 and vmbr3) connect? How viable is my approach to building a system at all?
Perhaps I did not plan everything correctly at all, but I would like to get a working system in exactly the configuration that I described above, because I want to have exactly divided subnets and control traffic between them (I'm talking about vmbr2 and vmbr3 networks).
I will be very grateful for any of your advice and your help…

 

[weehooey-bh]

Hi !

I have installed a PVE hypervisor on the physical server.It is connected to the enterprise's shared network through the vmbr0 virtual switch created by default.
An IP address associated with vmbr0 has been assigned.My idea was to create a virtual machine inside the PVE and make this VM a virtual router.
After creating the VM, I connected vmbr0 to it and assigned it the same IP address that was previously linked to the PVE management.
Then I created 3 more virtual switches (vmbr1, vmbr2, vmbr3) and added them to the virtual router.
vmbr1 – for communication with other hardware hypervisors (clustering)
vmbr2 – for server VMs
vmbr3 – for virtual workstationsI assigned each of them my own IP address of a separate subnet and enabled address forwarding (IP forward).

Now, traffic from the enterprise network enters the virtual router, from which it is routed via vmbr1 to the PVE management.And the rest (vmbr2 and vmbr3) will be used only for accessing virtual machines.And now the task has appeared to install a second (and in the future a third) PVE, connect PVE1 and PVE2 through one physical server port. Connect them to each other via a switchboard.
And vmbr2 and vmbr3 can also be used on the second PVE.
But here's the problem... PVE does not allow you to pass multiple virtual switches through one physical server port. Only 1 to 1.
I'm not a network technology specialist, so I'm asking for your help.I would like to make sure that all 3 networks are connected to the rest of the PVE hosts. If I combine two PVE hosts into a cluster and the connection between them will be only through vmbr1 (clustering), then will the others (vmbr2 and vmbr3) connect? How viable is my approach to building a system at all?
Perhaps I did not plan everything correctly at all, but I would like to get a working system in exactly the configuration that I described above, because I want to have exactly divided subnets and control traffic between them (I'm talking about vmbr2 and vmbr3 networks).
I will be very grateful for any of your advice and your help…

To confirm, you only have one physical network interface on the server? Getting at least one additional network port would make your life easier if you can.

Do you have a managed switch that supports VLANs? Do you have administrative access to that switch? If so, the best solution would be to use VLANs.

If you do not have a managed switch or access to it and only have one physical port per server, VXLANs would allow you to do this.

The best scenario would be physically separate network interfaces and a managed switch supporting VLANs.

 

[nightingale1981]

To confirm, you only have one physical network interface on the server? Getting at least one additional network port would make your life easier if you can.

Do you have a managed switch that supports VLANs? Do you have administrative access to that switch? If so, the best solution would be to use VLANs.

If you do not have a managed switch or access to it and only have one physical port per server, VXLANs would allow you to do this.

The best scenario would be physically separate network interfaces and a managed switch supporting VLANs.

Thank you.


There are 4 interfaces on the server. 1 - used to connect the server to the corporate network, 2 - used to manage PVE (if network 1 becomes unavailable). And there are two other ports that are free.

Yes, there is a managed switch with VLAN support. This is how I planned to connect the current PVE with the new PVE through this switch.

I just can't figure out how to do it right....

I need to make three vmbrs common for all PVE (vmbr1, vmbr2, vmbr3).

I ask you to give me advice and help, how can I do this correctly?

 

[weehooey-bh]

Please post the contents of your /etc/network/interfaces file. This will provide a clear picture of what you have right now.

 

[nightingale1981]

Please post the contents of your /etc/network/interfaces file. This will provide a clear picture of what you have right now.

Yes, of course.

______________________________________________________
PVE:
--------------------------------------------------------------------------

Linux pve 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64

auto lo
iface lo inet loopback

iface ens1f1 inet manual

iface enp3s0f0 inet manual

iface enp3s0f1 inet manual

iface ens1f0 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports ens1f1
bridge-stp off
bridge-fd 0
#Enterprise

auto vmbr1
iface vmbr1 inet static
address 100.60.3.2/24
gateway 100.60.3.1
bridge-ports enp3s0f0
bridge-stp off
bridge-fd 0
#MGMT PVE

auto vmbr2
iface vmbr2 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Servers

auto vmbr3
iface vmbr3 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Workstations

source /etc/network/interfaces.d/*

______________________________________________________
Virtual Router (Linux Debian)
--------------------------------------------------------------------------

auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens18
iface ens18 inet static
address 100.60.3.1/24


allow-hotplug ens19
iface ens19 inet static
address 100.20.20.1/24


allow-hotplug ens20
iface ens20 inet static
address 100.10.10.1/24

allow-hotplug ens21
iface ens21 inet static
address 192.168.28.150/24
gateway 192.168.28.1

 

[weehooey-bh]

Yes, of course.

______________________________________________________
PVE:
--------------------------------------------------------------------------

Linux pve 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64

auto lo
iface lo inet loopback

iface ens1f1 inet manual

iface enp3s0f0 inet manual

iface enp3s0f1 inet manual

iface ens1f0 inet manual

auto vmbr0
iface vmbr0 inet manual
bridge-ports ens1f1
bridge-stp off
bridge-fd 0
#Enterprise

auto vmbr1
iface vmbr1 inet static
address 100.60.3.2/24
gateway 100.60.3.1
bridge-ports enp3s0f0
bridge-stp off
bridge-fd 0
#MGMT PVE

auto vmbr2
iface vmbr2 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Servers

auto vmbr3
iface vmbr3 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#Workstations

source /etc/network/interfaces.d/*

______________________________________________________
Virtual Router (Linux Debian)
--------------------------------------------------------------------------

auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug ens18
iface ens18 inet static
address 100.60.3.1/24


allow-hotplug ens19
iface ens19 inet static
address 100.20.20.1/24


allow-hotplug ens20
iface ens20 inet static
address 100.10.10.1/24

allow-hotplug ens21
iface ens21 inet static
address 192.168.28.150/24
gateway 192.168.28.1

Thank you for sharing this.

You indicated you have four physical network interfaces on your server. I assume these are the ports:

• ens1f1
• enp3s0f0
• enp3s0f1
• ens1f0

It also appears that you have these subnets:

• 100.60.3.0/24 - Enterprise network? PVE management network?
• 100.10.10.0/24 - not sure
• 100.20.20.1/24 - Server VMs?
• 192.168.28.0/24 - Your workstations not in the cluster?

Is this close?

What VLANs do you currently use? What subnets are associated with those VLANs?

The virtual router you are planning to use, what IP address are you planning to use for its WAN interface (or the interface towards your ISP). What IP addresses are you planning to have on its other interfaces (e.g. the LAN interfaces)?

 

[nightingale1981]

Thank you for sharing this.

You indicated you have four physical network interfaces on your server. I assume these are the ports:

• ens1f1
• enp3s0f0
• enp3s0f1
• ens1f0

It also appears that you have these subnets:

• 100.60.3.0/24 - Enterprise network? PVE management network?
• 100.10.10.0/24 - not sure
• 100.20.20.1/24 - Server VMs?
• 192.168.28.0/24 - Your workstations not in the cluster?

Is this close?

What VLANs do you currently use? What subnets are associated with those VLANs?

The virtual router you are planning to use, what IP address are you planning to use for its WAN interface (or the interface towards your ISP). What IP addresses are you planning to have on its other interfaces (e.g. the LAN interfaces)?

And thank you.

Yes, it has 4 physical network interfaces.

ens1f0 and ens1f1 have 10 Gigabit ports. enp3s0f0 and enp3s0f1 are 1 GB ports

ens1f1 are connected to the corporate network. It is through this port that the connection to the virtual router is made. Its IP address is 192.168.28.0/24

The remaining physical ports are not in use yet.
I have added only one physical port enp3s0f0 to the PVE for access to the PVE management (in case of termination of access from the corporate network). Its network is 100.60.3.0/24 (it is connected to vmbr1). I would like to use this network to connect to new PVE through a physical switch. The virtual router in PVE has been assigned the address 100.60.3.1., and the address of the PVE itself is 100.60.3.2. In the future, new PVE addresses will be assigned 100.60.3.3 and 100.60.3.4 accordingly. And I would like to allocate one of the free server ports for connection (for example, enp3s0f0).
100.10.10.0/24 – this network is connected to vmbr2. It is planned to use it for the first type of virtual machines (for example, workstations).
200.20.20.0/24 is a network connected to vmbr3. It is planned to use it for the second type of virtual machines (for example, servers).

Like this.

I use virtual networks:
vmbr0 (192.168.28.0/24), vmbr1 (100.60.3.0/24), vmbr2 (100.10.10.0/24), vmbr3 (200.20.20.0/24)

The WAN address is 192.168.28.150 (but it won't work as a NAT, it will work as an internetwork router).

My problem is that I cannot redirect multiple virtual networks through one physical server port to connect to another PVE server. Only 1 to 1.

Or I will have to abandon such a scheme and use only the vmbr1 network.

But then it makes no sense to divide virtual networks into segments and route routes between them through a virtual router using several servers with PVE.

If I were going to use only one physical server with PVE, then everything would work for me and this problem would not arise at all, but I want to be able to horizontally expand the test system that I am currently planning.

Maybe using Open vSwitch technology helped me deal with this, but I don't really understand it.

 

[weehooey-bh]

And thank you.

Yes, it has 4 physical network interfaces.

ens1f0 and ens1f1 have 10 Gigabit ports. enp3s0f0 and enp3s0f1 are 1 GB ports

ens1f1 are connected to the corporate network. It is through this port that the connection to the virtual router is made. Its IP address is 192.168.28.0/24

The remaining physical ports are not in use yet.
I have added only one physical port enp3s0f0 to the PVE for access to the PVE management (in case of termination of access from the corporate network). Its network is 100.60.3.0/24 (it is connected to vmbr1). I would like to use this network to connect to new PVE through a physical switch. The virtual router in PVE has been assigned the address 100.60.3.1., and the address of the PVE itself is 100.60.3.2. In the future, new PVE addresses will be assigned 100.60.3.3 and 100.60.3.4 accordingly. And I would like to allocate one of the free server ports for connection (for example, enp3s0f0).
100.10.10.0/24 – this network is connected to vmbr2. It is planned to use it for the first type of virtual machines (for example, workstations).
200.20.20.0/24 is a network connected to vmbr3. It is planned to use it for the second type of virtual machines (for example, servers).

Like this.

I use virtual networks:
vmbr0 (192.168.28.0/24), vmbr1 (100.60.3.0/24), vmbr2 (100.10.10.0/24), vmbr3 (200.20.20.0/24)

The WAN address is 192.168.28.150 (but it won't work as a NAT, it will work as an internetwork router).

My problem is that I cannot redirect multiple virtual networks through one physical server port to connect to another PVE server. Only 1 to 1.

Or I will have to abandon such a scheme and use only the vmbr1 network.

But then it makes no sense to divide virtual networks into segments and route routes between them through a virtual router using several servers with PVE.

If I were going to use only one physical server with PVE, then everything would work for me and this problem would not arise at all, but I want to be able to horizontally expand the test system that I am currently planning.

Maybe using Open vSwitch technology helped me deal with this, but I don't really understand it.

Okay. Thanks. That helps to provide a clearer picture. My understanding from what you have provided:

NICs

• ens1f0 - 1G - unassigned
• ens1f1 - 1G - connected to the corporate network
• enp3s0f0 - 10G - unassigned
• enp3s0f1 - 10G - unassigned

Networks

• 192.168.28.0/24 - Corporate network
• 10.60.3.0/24 - PVE management network
• 100.10.10.0/24 - Workstations
• 200.20.20.0/24 - Servers

Note: I assume you know and changed the addresses to anonymize them, but in case you do not know, you should not use 100.10.10.0/24 and 200.20.20.0/24. These subnets are publicly routable and owned by others (e.g. Verizon owns 100.10.10.0/24).

First, Open vSwitch will not add any benefit and will not help you achieve your goals.

My suggestion would be something like the following based on what you have told me about your objectives:

Interfaces

• ens1f0
  • Reserve for a dedicated Corosync link when you add additional PVE nodes.
  • When you do use it, do not use a bridge. Just put the IP address on the interface.
• ens1f1
  • Leave it on vmbr0
  • Do not put any IP addresses on this interface or vmbr0
• enp3s0f0
  • Only used for PVE traffic
  • Put it on vmbr1
  • Put the PVE host IP address on the bridge
  • Connect this to a dedicated switch or a VLAN access port with a VLAN defined for only PVE management traffic. The switch or VLAN should not go anywhere other than to the PVE management interfaces.
• enp3s0f1
  • Put it on vmbr3
  • Reserve for guest traffic in the cluster
  • In the switch, set it to accept tagged traffic (i.e. trunk).
  • No IP address on this interface
• Remove:
  • ens18, ens19, ens20, ens21
  • I understand that you created these, and they are not physical interfaces. If this is correct, remove them.

Bridges

It may be helpful to know two things:

1. Bridges act like switches (so you can think "switch" when trying to understand their behaviour) and
2. If you put an IP address on an interface, bond, or bridge in the interfaces file, you are giving the PVE host access to that, and anything connected to it will have access to the PVE host.

• vmbr0
  • This connects your virtual router to your corporate network.
  • The only thing connected to it in the interfaces file is ens1f1.
• vmbr1
  • This bridge will be your PVE management bridge.
  • You will hang the PVE management IP address on this bridge from the 10.60.3.0/24 subnet.
• vmbr3
  • This bridge will be used for your guest traffic (all of it, workstations, servers, everything).
  • Do not put an IP address on it.

Configuration in PVE

• In the SDN
  • Create a VLAN Zone on vmbr3
  • Create a Vnet for your workstations, for example, call it VLAN100 and use the VLAN tag 100.
  • Create a Vnet for your servers; for example, call it VLAN200 and use the VLAN tag 200.
• Virtual router
  • One network interface will be on vmbr0, this will be for your "WAN" and will connect it to your corporate network.
  • vmbr1 on another interface used for PVE management.
  • Then, you will create an interface for each guest VLAN and assign the interface to the SDN Vnet for that VLAN (the Vnets will appear as bridges).
• Guests
  • When you create the guest VM, you give it a NIC and assign it to the appropriate bridge (one of the SDN Vnets).

Summary

Traffic will flow in vmbr0 and only go to your virtual router, which will route the traffic as necessary.

If the traffic is for the PVE host(s), it will flow out of the router interface connected to vmbr1. From vmbr1, it will go to the host or out of the physical interface over the switch, and to the other host(s).

If the traffic is for a guest, it will flow out the appropriate interface connected to an SDN Vnet (bridge) and get a VLAN tag. In the Vnet bridge, it will be able to flow to other guests with the same tag or flow out to vmbr3. From vmbr3, it will flow out to the switch with its tag to the other PVE hosts, where it will be able to access the guests connected to the same Vnet (i.e. with the same VLAN tag).

Of course, the traffic will be able to flow in the opposite direction, assuming you do not have any ACLs or firewall rules on the virtual router.

Hopefully, this gives you an idea of how to structure things. There are other ways you could do this, but I tried to make it fit with your goals.

Let me know if you have any questions.

 

[nightingale1981]

thank you very much!

I was puzzled and had questions.

You suggested the following:

Remove:

• ens18, ens19, ens20, ens21
• I understand that you created these, and they are not physical interfaces. If this is correct, remove them.

If I delete ens18 (this is the gateway for the 100.60.3.0/24 network). Without it, I will not be able to access the network to manage PVE from the corporate network. And I want to keep this opportunity.
ens21 is also a virtual interface looking towards the corporate network. Deleting it will also make it impossible to connect from it.
I can delete ens19 (it is vbmr2).
ens20 – I will also delete it. But then how will traffic be routed between the networks (between 100.10.10.0/24 and 200.20.20.0/24)?
After all, there should be networks vbmr2 (100.10.10.1) and vbmr3 (200.20.20.1) on the virtual router.
I meant that all routing would go through a virtual router.
So far, ignorance of the principles of SDN prevents me from understanding this.

And you also wrote:

Then, you will create an interface for each guest VLAN and assign the interface to the SDN Vnet for that VLAN (the Vnets will appear as bridges).

I can only assign an interface – a network interface (for example, an existing vbmr3 to a new or existing VM). Unfortunately, I could not understand this point.

I think it's not going to be easy.

 

[weehooey-bh]

If I delete ens18 (this is the gateway for the 100.60.3.0/24 network). Without it, I will not be able to access the network to manage PVE from the corporate network. And I want to keep this opportunity.

Your virtual router will be your gateway on 10.60.3.0/24. The traffic from your PVE subnet will go to the IP address on your virtual router to be routed to your corporate network. ens18 is not working in your configuration as you expect.

ens21 is also a virtual interface looking towards the corporate network. Deleting it will also make it impossible to connect from it.

You indicated that you want your traffic from the corporate network to go through your virtual router. It will come in ens1f1 through vmbr0 and reach your virtual router's "WAN" interface. You will assign an IP address in your virtual router in your corporate network.

ens20 – I will also delete it. But then how will traffic be routed between the networks (between 100.10.10.0/24 and 200.20.20.0/24)?

It will be routed by your virtual router.

After all, there should be networks vbmr2 (100.10.10.1) and vbmr3 (200.20.20.1) on the virtual router.
I meant that all routing would go through a virtual router.

Yes, traffic between the four subnets will be handled by your virtual router.

I can only assign an interface – a network interface (for example, an existing vbmr3 to a new or existing VM). Unfortunately, I could not understand this point.

What are you using for your virtual router?

 

[nightingale1981]

Debian 12