Help understanding routed networking for publically hosted PVE?

[MrGeezer]

Hi - wondering if anyone can tell me whether I have misunderstood how I am supposed to be doing routed networking. I have a dedicated server with proxmox and 4 public IP's. I've been using 2 of them for a long time without any issues but I am now attempting to use the final one as the public IP for a VM running PfSense as a router for several further VM's on a private LAN and I am having many problems with PfSense dropping connections after allowing them to initiate, not allowing clients on its LAN side to do DNS lookups on the WAN side etc. One of the other 2 IP's has also been set up this way and although I thought it was fine, it is not used for much apart from backup data and upon adding some additional guests to its LAN I note that it is actually seeming to have the same problems, leading me to believe I may have misunderstood the process all along.

I have 4 public IP's and one useable NIC. Say public IP's are 10.10.10.30 (gateway 10.10.10.1), and 10.10.10.40, 10.10.10.41 and 10.10.10.42 and the NIC name is enp4s0f1

My /etc/network/interfaces is setup as

auto enp4s0f1
iface enp4s0f1 inet static
        address 10.10.10.30/32
        gateway 10.10.10.1
        hwaddress ab:cd:ef:01:02:03
        pointopoint 10.10.10.1
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/enp3s0f0/proxy_arp

iface (OTHER NIC FOR KVM NOT AVAILABLE) inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.30/32
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        up ip route add 10.10.10.40/32 dev vmbr0
        up ip route add 10.10.10.41/32 dev vmbr0
        up ip route add 10.10.10.42/32 dev vmbr0

I then set the NIC on my long working VM to be connected to vmbr0 and the settings as:

IP: 10.10.10.40 - default gateway = 10.10.10.30

However I have set the WAN side NIC on my PfSense VM to be:

IP 10.10.10.41 - default gateway = 10.10.10.30

and am experiencing the problems described above. Although it does sucessfully connect to the internet, there are all kinds of intermittent problems. When I look in the logs there are packets being dropped by the firewall to and from the other IPs, 10.10.10.30 and 10.10.10.40 etc.

I am wondering if I have misunderstood how I should have set this up? The diagram at https://pve.proxmox.com/pve-docs/images/default-network-setup-routed.svg suggests I have got the right idea so perhaps it's PfSense that I have configured wrong?

Any clues for what to do or where to go with my reasearching would be much appreciated.

Tia

 

[dakralex]

Have you tried to change the enp3s0f0 in the line post-up echo 1 > /proc/sys/net/ipv4/conf/enp3s0f0/proxy_arp to the actual network card's name enp4s0f1? It could be that pfSense or your hosting provider is strict about which MAC addresses are on the network or are using your IP addresses and therefore you should proxy them.