Hello All,
I've run into an interesting problem and for the last few days, I tried to figure out what the root cause of this phenomenon could be.
As you read in the title I try to point networks to the Pfsense gateway. this works great as long as the Test-VM and Pfsense are on the same machine.
And it seems that it does not matter if I use standard Linux bridges or the OVS (same behavior). The routing between the nodes works (I can manage the firewall and Proxmox servers no matter where the Test-VM is) but the internet gateway fails once the signal has to change hosts.
My guess would be that I somehow missed a crucial configuration in Proxmox which hinders the path back to the original node and VM.
My guess goes to Proxmox since I see in the firewall log that the rule is checked and the outbound traffic allowed. (At the moment every traffic from any local network to any other local network and the WAN is permitted and l have logs enabled)
Do you have any idea where my problem comes from?
I thank you in advance for your help
The setup:
3 Nodes with the following networks
- 2 1 GBE NICs for direct Corosync to the other nodes (192.168.4.11/24)
- 2 10 GBE NICs for inter-cluster communication 1 to each node (10.0.0.0/24)
- 2 10 GBE NICs for Ceph1 to each node (192.168.6.11/24)
- 1 Exit port to the main switch (management interface vlan116: 192.168.5.0/24)
My Network config from the first node (but basically identical everywhere):
The firewall has 3 virtual NICs
- "WAN"
- LAN1 (192.168.5.1)
- LAN2 (10.0.0.1)
I've run into an interesting problem and for the last few days, I tried to figure out what the root cause of this phenomenon could be.
As you read in the title I try to point networks to the Pfsense gateway. this works great as long as the Test-VM and Pfsense are on the same machine.
And it seems that it does not matter if I use standard Linux bridges or the OVS (same behavior). The routing between the nodes works (I can manage the firewall and Proxmox servers no matter where the Test-VM is) but the internet gateway fails once the signal has to change hosts.
My guess would be that I somehow missed a crucial configuration in Proxmox which hinders the path back to the original node and VM.
My guess goes to Proxmox since I see in the firewall log that the rule is checked and the outbound traffic allowed. (At the moment every traffic from any local network to any other local network and the WAN is permitted and l have logs enabled)
Do you have any idea where my problem comes from?
I thank you in advance for your help
The setup:
3 Nodes with the following networks
- 2 1 GBE NICs for direct Corosync to the other nodes (192.168.4.11/24)
- 2 10 GBE NICs for inter-cluster communication 1 to each node (10.0.0.0/24)
- 2 10 GBE NICs for Ceph1 to each node (192.168.6.11/24)
- 1 Exit port to the main switch (management interface vlan116: 192.168.5.0/24)
My Network config from the first node (but basically identical everywhere):
auto lo
iface lo inet loopback
iface enp3s0f0 inet manual
iface enp3s0f1 inet manual
iface enp4s0f0 inet manual
iface enp4s0f1 inet manual
iface ens1f0 inet manual
iface ens1f1 inet manual
iface ens2f0 inet manual
iface ens2f1 inet manual
allow-vmbr1 vlan3
iface vlan3 inet static
address 10.0.0.4/24
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_options tag=3
auto bond0
iface bond0 inet static
address 192.168.4.11/24
bond-slaves enp3s0f1 enp4s0f0
bond-miimon 100
bond-mode broadcast
#Corosync
auto bond1
iface bond1 inet static
address 192.168.6.11/24
bond-slaves ens1f0 ens1f1
bond-miimon 100
bond-mode broadcast
#Ceph_Net
allow-vmbr1 bond2
iface bond2 inet manual
ovs_bonds ens2f0 ens2f1
ovs_type OVSBond
ovs_bridge vmbr1
ovs_options lacp=active bond_mode=balance-tcp
#VM_NET_Ports
auto vmbr0
iface vmbr0 inet manual
bridge-ports enp3s0f0 enp4s0f1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 116
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '192.168.5.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.5.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
#Public_Port
auto vmbr0.116
iface vmbr0.116 inet static
address 192.168.5.2/24
gateway 192.168.5.1
#ADMIN
allow-ovs vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond2 vlan3
#ovs connection
post-up ovs-vsctl add-port vmbr1 gre0 -- set interface gre0 type=gre options:remote_ip='10.0.0.5'
post-up ovs-vsctl add-port vmbr1 gre1 -- set interface gre1 type=gre options:remote_ip='10.0.0.6'
#VM_Network
The firewall has 3 virtual NICs
- "WAN"
- LAN1 (192.168.5.1)
- LAN2 (10.0.0.1)
