[SOLVED] Proxmox behind OPNsense Hardware Firewall; Routing Problem

[gProxiA]

Hi,

here is my Setup:

Hardware Firewall (Opnsense):

• 3 WAN Interfaces with Public IPs and Upstream Gateways
• LAN Interface; Network: 10.0.0.0/16; IP: 10.0.0.10
• DMZ Interface; 192.168.0.0/16; IP: 192.168.0.10
• 2 Virtual OpenVPN Interfaces;

Proxmox Server:
I have two NICs, each of them is wired to the Hardware Firewall.

• eno1: Connected to LAN Interface from the FW
• eno2: Connected to DMZ Interface from the FW
• vmbr0: Bridge connected to eno1; IP: 10.0.0.3; GW: 10.0.0.10 (LAN Interface of the FW act as a Gateway)
• vmbr1: Bridge connected to eno2; IP: 192.168.0.3; no GW, because it is only possible to have one Gateway

Problem:
The VMs and LXCs are able to reach the Internet, and are also accessible from the Clients. But I can't ping my Firewall (10.0.0.10), neither from the Proxmox Host or the VMs. But I can ping from my Gateway back to the Proxmox Host. I investigated this problem with traceroute, and here is what i noticed:

Traceroute from Proxmox Host to Firewall:
traceroute 10.0.0.10:
1: 10ms 13ms 12ms (firewall.domain.com) 10.0.0.10
2: 45ms 23ms 32ms <Public IP, of WAN Interface>
3: * * *
4 * * *

I tried it with different Interfaces, but no success. When i am connected via VPN, i am able to ping the Firewall. I am also able to access the Vms and LXs but not the Proxmox host.

The ICMP packet looked like this:
SOURCE: <public IP>
DESTINATION: 10.0.0.10

I don't understand why my Firwall assign an internal packet with an public IP and routing it outside?
Normally it should look like this: Soure: 10.0.0.3 destination 10.0.0.10

does anyone have any idea what I did wrong? Thank you!

 

[eoinkim]

I think your firewall is not configured with NAT?

 

[gProxiA]

I think your firewall is not configured with NAT?

I have configured Pure NAT.

 

[eoinkim]

Hmm... strange. So, NAT is configured but still routing externally? Your traceroute should end with one hop, isn't it correct?
What about the Proxmox node itself? Does it have any firewall setup?

 

[gProxiA]

The strange thing is, that an internal Packet got assigned the public ip as destination IP...
Theoretically traceroute should end after one hop.
1. x ms x ms x ms <IP of firewall>

Proxmox Firewall is configured. But i think this has nothing todo with the routing problem. ICMP is on each layer allowed.
Today I have completely rebuilt my setup in a virtual environment and behold, everything works.

Here is my Routing Table on Firewall:
see attachments

 

[gProxiA]

Hi,

a missing firewall rule caused this problem. If you have configured Multi WAN, all traffic gets routed via the WAN Interfaces.
You have to set a rule for the internal traffic to use the internal/default gateway.