We have set up a couple of nodes in the way the diagram indicates.
All our VM's are on a bridged NIC that is also shared by the pfSense VM LAN port. Let's for example say they are on 192.168.100.0/24 for the VLAN in question.
Since there are multiple tenants on this systems, we have created SDN's for them, each on a different VLAN. So client1's VLAN is VLAN20 for instance, client2 is VLAN21 and so forth, with each client VLAN also set up as a separate port on the pfSense VMs. So we simply use an alias for each VLAN and thus we see in pfSense "Joe Bloggs" and "Winnie the Poo" as client networks, representing the client's VLANs. The default gateway on each client/guest VM is the address of the port on the pfSense router of that particular VLAN.
To allow failover we have set up CARP to have a floating IP address for each VLAN for redundancy.
Depending on the client's requirements we either NAT from the WAN gateway of pfSense to their VLAN, or we just allow the traffic through unNAT'ed and assign a public IP address to their client VM on their VLAN.
This all works really well.
Now a particular client has the need to have their own Sophos firewall on their VLAN. (They use Sophos on their client sites and want to configure their own VPN connections with this). So we created a VM with a Sophos image installed and configured with a port for each VLAN that should terminate in the Sophos instead of pfSense. The WAN port on the Sophos connects to a non-NAT'ed port on pfSense with a public ip address. (As we need more public addresses on the Sophos, we wil simply add them). The client/guest machines now simply need to have their default gateway changed to the address of the Sophos VLAN port to route their traffic through the Sophos instead of pfSense.
e.g. Client VM 3, on VLAN20, with ip address range 192.168.100.0/24. has an address 192.168.100.254 on pfSense and 192.168.100.10 on the Sophos. So if we set the default gateway on VM3 to 192.168.100.10, the traffic will go via the sophos, and if we set to 192.168.100.254 it will go via pfSense directly. This is where the problem lies.
The traffic from a client machine always goes via 192.168.100.254, regardless of which default gateway is set on the client/guest VM. If we add static routes to the client, then that clients' traffic actually exits via the specified gateway.
Why, is a mystery to us at this stage.
This is still Proxmox 7 (upgrades are being planned as we speak) with SDN. We have added the subnets in the SDN VNets, but understood that these are not used as they are in later version (i.e. 8.1). Is this true and could this be the source of the problem? Surely the gateway setting on a client VM is definitive regardless of the what is set up in the SDN or is it?
It there something else that can be causing this?
All our VM's are on a bridged NIC that is also shared by the pfSense VM LAN port. Let's for example say they are on 192.168.100.0/24 for the VLAN in question.
Since there are multiple tenants on this systems, we have created SDN's for them, each on a different VLAN. So client1's VLAN is VLAN20 for instance, client2 is VLAN21 and so forth, with each client VLAN also set up as a separate port on the pfSense VMs. So we simply use an alias for each VLAN and thus we see in pfSense "Joe Bloggs" and "Winnie the Poo" as client networks, representing the client's VLANs. The default gateway on each client/guest VM is the address of the port on the pfSense router of that particular VLAN.
To allow failover we have set up CARP to have a floating IP address for each VLAN for redundancy.
Depending on the client's requirements we either NAT from the WAN gateway of pfSense to their VLAN, or we just allow the traffic through unNAT'ed and assign a public IP address to their client VM on their VLAN.
This all works really well.
Now a particular client has the need to have their own Sophos firewall on their VLAN. (They use Sophos on their client sites and want to configure their own VPN connections with this). So we created a VM with a Sophos image installed and configured with a port for each VLAN that should terminate in the Sophos instead of pfSense. The WAN port on the Sophos connects to a non-NAT'ed port on pfSense with a public ip address. (As we need more public addresses on the Sophos, we wil simply add them). The client/guest machines now simply need to have their default gateway changed to the address of the Sophos VLAN port to route their traffic through the Sophos instead of pfSense.
e.g. Client VM 3, on VLAN20, with ip address range 192.168.100.0/24. has an address 192.168.100.254 on pfSense and 192.168.100.10 on the Sophos. So if we set the default gateway on VM3 to 192.168.100.10, the traffic will go via the sophos, and if we set to 192.168.100.254 it will go via pfSense directly. This is where the problem lies.
The traffic from a client machine always goes via 192.168.100.254, regardless of which default gateway is set on the client/guest VM. If we add static routes to the client, then that clients' traffic actually exits via the specified gateway.
Why, is a mystery to us at this stage.
This is still Proxmox 7 (upgrades are being planned as we speak) with SDN. We have added the subnets in the SDN VNets, but understood that these are not used as they are in later version (i.e. 8.1). Is this true and could this be the source of the problem? Surely the gateway setting on a client VM is definitive regardless of the what is set up in the SDN or is it?
It there something else that can be causing this?
Attachments
Last edited: