Proxmox networking and layer 3 routing

[Overfill4993]

The title contains what I am trying to achieve.

My setup contains of a few virtual machines, most of them are in the same VLAN I have created for servers (I will refer to that as VLAN 100). One of my VM that hosts most of my services for home devices is in different VLAN (VLAN 2).

The setup works just fine and my firewall can route 1Gbe with relative ease. However I just feel it is a slight misuse of resources to route VM-VM traffic through firewall instead of routing the traffic inside Proxmox. Unfortunatelu I have not enough experience with Proxmox to know how to solve my problem. OVS seems to have Layer 3 capabilities, but I haven't found a noob-friendly guide on setting that up. Also I'm not actually sure if OVS is the go-to-solution for my problem.

TL;DR: What is the best way to achieve Layer 3 routing inside Proxmox for virtual machines inside different VLANs?

 

[Spoonman2002]

https://www.networkshinobi.com/proxmox-ve-with-open-vswitch/

This website helped me a lot with OVS.

 

[Overfill4993]

https://www.networkshinobi.com/proxmox-ve-with-open-vswitch/

This website helped me a lot with OVS.

I got my setup working with that guide, thanks!

Something weird is still happening with routing and I have trouble understanding it. I have a original Linux bridge (vmbr0 from installation) and it is used for management. I left that untouched because I want my VM traffic on a different port. I destroyed the earlier VM-network bridge (vmbr1) and started to use the port it was using (enp131s0).

I created OVS Bridge (new vmbr1) and OVS IntPorts for my VLANs. Everything went quite smooth and as instructed. Now everything is working pretty much the same as earlier, but the core problem with routing has not solved itself. The traffic between VLAN100 and VLAN2 are still being sent to the router instead of them being routed at the host.

Could the cause be that I still have the original vmbr0 running for management?

 

[Spoonman2002]

I think the website wrote that you can delete the original vmbr0 Linux Bridge and create the new OVS Bridge.
My personal PVE host #1 (I have 3) has an OVS Bridge with several VLANs (IntPorts).

You can set the IP address for management on the OVS Bridge, or.......
you can give an IntPort (VLAN) an IP address for management.
The IntPort IP address has the advantage that it is already a dedicated VLAN.

This is how I unsterstand OVS, not saying my config is correct or an example for others to follow.

 

[Overfill4993]

I think the website wrote that you can delete the original vmbr0 Linux Bridge and create the new OVS Bridge.
My personal PVE host #1 (I have 3) has an OVS Bridge with several VLANs (IntPorts).

You can set the IP address for management on the OVS Bridge, or.......
you can give an IntPort (VLAN) an IP address for management.
The IntPort IP address has the advantage that it is already a dedicated VLAN.

This is how I unsterstand OVS, not saying my config is correct or an example for others to follow.

I just recreated the management interface as IntPort. Now everything is configured using one OVS Bridge. The problem persisted in cross VLAN traffic from VM to VM.

Did some reading and the problem was that the OVS Bridge didn't have sufficient information to do Layer 3 routing. It had VMs in different VLANs, but only one gateway as an address from management-port. I edited my two VLAN IntPorts to have addresses (write directly to IPv4/CIDR-line or write to the /etc/network/interfaces corresponding VLAN lines) and after this I set net.ipv4.ip_forward = 1 in sysctl.conf. Stuff started to work as I hoped. Pictures explain this better so there's a couple.



Now the VM-VM traffic doesn't overwork my router. There is a downside though. I now have intermittent connectivity issues with SSH to Proxmox. This no doubt has something to do with the fact that there are now multiple gateway addresses configured in the bridge. I will probably split the management back to different bridge and see if that fixes this issue.

If anyone is interested I used this site for my information.

 

[Spoonman2002]

I just recreated the management interface as IntPort. Now everything is configured using one OVS Bridge. The problem persisted in cross VLAN traffic from VM to VM.

Did some reading and the problem was that the OVS Bridge didn't have sufficient information to do Layer 3 routing. It had VMs in different VLANs, but only one gateway as an address from management-port. I edited my two VLAN IntPorts to have addresses (write directly to IPv4/CIDR-line or write to the /etc/network/interfaces corresponding VLAN lines) and after this I set net.ipv4.ip_forward = 1 in sysctl.conf. Stuff started to work as I hoped. Pictures explain this better so there's a couple.
View attachment 52490
View attachment 52491

Now the VM-VM traffic doesn't overwork my router. There is a downside though. I now have intermittent connectivity issues with SSH to Proxmox. This no doubt has something to do with the fact that there are now multiple gateway addresses configured in the bridge. I will probably split the management back to different bridge and see if that fixes this issue.

If anyone is interested I used this site for my information.

Not sure configuring multiple gateways on Proxmox host is THE way to go.
(I always hear people saying in this forum that only 1 gateway is allowed).
You now have configured Proxmox as a "router" with net.ipv4.ip_forward=1
That solution is what you asked for in your startpost, so you solved your own question, great.

Personally I would just let my physical router device handle ALL VLAN traffic, that's what it's purpose is for.
And let Proxmox be a hypervisor.
But then again, you wouldn't have started this thread in the first place.

 

[Overfill4993]

Not sure configuring multiple gateways on Proxmox host is THE way to go.
(I always hear people saying in this forum that only 1 gateway is allowed).
You now have configured Proxmox as a "router" with net.ipv4.ip_forward=1
That solution is what you asked for in your startpost, so you solved your own question, great.

Personally I would just let my physical router device handle ALL VLAN traffic, that's what it's purpose is for.
And let Proxmox be a hypervisor.
But then again, you wouldn't have started this thread in the first place.

Yeah, multiple gateways is not the solution. Connectivity becomes very unstable after a while, I have to figure out something else. Modifying sysctl.conf was actually not necessary now that I retested some scenarios, layer 3 routing worked fine without it.

In my, admittedly amateur mind, routing VLAN traffic to router in my use case seems like a suboptimal solution. I agree with you when the data flows between devices in same VLAN or cross VLANs. But when the traffic goes from one server, to router and back to same physical machine, the routing just seems like a hindrance to me.

I feel like there are a few questions I need answers to:
1. Am I trying to achieve something that is either not supported or not encouraged?
2. Am I just not proficient enough with OVS?

What bugs me is that I know that OVS can do layer 3, but I just don't know where to go from here. I got it to "work", but not in a sustainable way. The source I am using is 10 years old and I rather not follow it. Back to studying OVS I guess.

 

[Spoonman2002]

Yeah, multiple gateways is not the solution. Connectivity becomes very unstable after a while, I have to figure out something else. Modifying sysctl.conf was actually not necessary now that I retested some scenarios, layer 3 routing worked fine without it.

In my, admittedly amateur mind, routing VLAN traffic to router in my use case seems like a suboptimal solution. I agree with you when the data flows between devices in same VLAN or cross VLANs. But when the traffic goes from one server, to router and back to same physical machine, the routing just seems like a hindrance to me.

I feel like there are a few questions I need answers to:
1. Am I trying to achieve something that is either not supported or not encouraged?
2. Am I just not proficient enough with OVS?

What bugs me is that I know that OVS can do layer 3, but I just don't know where to go from here. I got it to "work", but not in a sustainable way. The source I am using is 10 years old and I rather not follow it. Back to studying OVS I guess.

What if you virtualise a router like pfSense/OPNsense and run it as a vm on your Proxmox host?
Is that a few bottlenecks less in the networking chain for you?

 

[Overfill4993]

What if you virtualise a router like pfSense/OPNsense and run it as a vm on your Proxmox host?
Is that a few bottlenecks less in the networking chain for you?

The forbidden router you say?

I have considered it, but it makes my network too centralized and has major downsides. For performance that would be ideal solution, but I prefer having a separate router.