VM only uses Pfsense gateway when on the same Server(Node)

Belliger

New Member
Dec 23, 2020
3
0
1
30
Hello All,

I've run into an interesting problem and for the last few days, I tried to figure out what the root cause of this phenomenon could be.
As you read in the title I try to point networks to the Pfsense gateway. this works great as long as the Test-VM and Pfsense are on the same machine.
And it seems that it does not matter if I use standard Linux bridges or the OVS (same behavior). The routing between the nodes works (I can manage the firewall and Proxmox servers no matter where the Test-VM is) but the internet gateway fails once the signal has to change hosts.

My guess would be that I somehow missed a crucial configuration in Proxmox which hinders the path back to the original node and VM.
My guess goes to Proxmox since I see in the firewall log that the rule is checked and the outbound traffic allowed. (At the moment every traffic from any local network to any other local network and the WAN is permitted and l have logs enabled)

Do you have any idea where my problem comes from?

I thank you in advance for your help



The setup:
3 Nodes with the following networks
- 2 1 GBE NICs for direct Corosync to the other nodes (192.168.4.11/24)
- 2 10 GBE NICs for inter-cluster communication 1 to each node (10.0.0.0/24)
- 2 10 GBE NICs for Ceph1 to each node (192.168.6.11/24)
- 1 Exit port to the main switch (management interface vlan116: 192.168.5.0/24)

My Network config from the first node (but basically identical everywhere):
auto lo iface lo inet loopback iface enp3s0f0 inet manual iface enp3s0f1 inet manual iface enp4s0f0 inet manual iface enp4s0f1 inet manual iface ens1f0 inet manual iface ens1f1 inet manual iface ens2f0 inet manual iface ens2f1 inet manual allow-vmbr1 vlan3 iface vlan3 inet static address 10.0.0.4/24 ovs_type OVSIntPort ovs_bridge vmbr1 ovs_options tag=3 auto bond0 iface bond0 inet static address 192.168.4.11/24 bond-slaves enp3s0f1 enp4s0f0 bond-miimon 100 bond-mode broadcast #Corosync auto bond1 iface bond1 inet static address 192.168.6.11/24 bond-slaves ens1f0 ens1f1 bond-miimon 100 bond-mode broadcast #Ceph_Net allow-vmbr1 bond2 iface bond2 inet manual ovs_bonds ens2f0 ens2f1 ovs_type OVSBond ovs_bridge vmbr1 ovs_options lacp=active bond_mode=balance-tcp #VM_NET_Ports auto vmbr0 iface vmbr0 inet manual bridge-ports enp3s0f0 enp4s0f1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 116 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '192.168.5.0/24' -o vmbr0 -j MASQUERADE post-down iptables -t nat -D POSTROUTING -s '192.168.5.0/24' -o vmbr0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 #Public_Port auto vmbr0.116 iface vmbr0.116 inet static address 192.168.5.2/24 gateway 192.168.5.1 #ADMIN allow-ovs vmbr1 iface vmbr1 inet manual ovs_type OVSBridge ovs_ports bond2 vlan3 #ovs connection post-up ovs-vsctl add-port vmbr1 gre0 -- set interface gre0 type=gre options:remote_ip='10.0.0.5' post-up ovs-vsctl add-port vmbr1 gre1 -- set interface gre1 type=gre options:remote_ip='10.0.0.6' #VM_Network

The firewall has 3 virtual NICs
- "WAN"
- LAN1 (192.168.5.1)
- LAN2 (10.0.0.1)
 
Can noone help me?
I can't be the onlyone having a cluster with a single firewall oO
 
If you want to use the firewall as standard gateway for the VM, the firewall must be in the same L2 network (or VLAN) as the VM. GRE is a L3 encapsulation protocol, not a L2. The simplest solution from my point of view would be to use 802.1q VLAN tagging between your nodes (only for the inter-cluster communication). See our network config documentation for more details.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!