VM can't connect to own Public IP

[dsrvlaga]

Hey Everyone,

I'm stuck with a problem and I can't find out how to solve it. I've tried to find it on the internet but till now even that didn't help.

I currently have a proxmox server with a vm that runs plesk (webserver hosting panel). I managed to correctly add a network bridge that my vm is connected to and added port forwarding using iptables. So my VM has internet and can be accessed with a public ip. I also configured the firewall and that works (if i'm correct) also fine.

But now comes the issue. The VM can ping/curl to the private ip and also to 'any' other public ip but, it can't ping or curl to the public ip (of the server). So for example my domain wants to use the api of a subdomain (both on the same vm and have the same internal ip), thus it uses for example curl auth.domain.com and then it gives a timeout error. How can I solve this issue.

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
        address XX.XX.XX.XX
        netmask 255.255.255.0
        gateway XX.XX.XX.
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        
auto vmbr1
iface vmbr1 inet static
        address 10.10.10.1/24
        netmask 255.255.255.0
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

        # Port Forwarding
        post-up   iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 10.10.10.2:443
        post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 10.10.10.2:443

        post-up   iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.10.10.2:80
        post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 10.10.10.2:80

/proc/sys/net/ipv4/ip_forward
contains 1

qm config <VMID>

boot: order=scsi0;ide2;net0
cores: 20
ide2: none,media=cdrom
machine: q35
memory: 16384
meta: creation-qemu=6.1.0,ctime=1647968362
name: website
net0: virtio=06:47:C1:CB:F5:8F,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,size=1200G
scsihw: virtio-scsi-pci
smbios1: uuid=2ce5b6f1-7ee5-457b-8fa3-ac8544abbcd4
sockets: 1
vga: qxl
vmgenid: e2d04b92-fd80-4c0d-87e5-665554001c71

ip r on the vm

default via 10.10.10.1 dev enp6s18 proto static metric 100
10.10.10.0/24 dev enp6s18 proto kernel scope link src 10.10.10.2 metric 100

 

[gurubert]

The DNAT rules will not be active in this case as the packet will not come in via vmbr0 (from the Internet) but vmbr1 (from the VM).
Even if that would be corrected you will not get the reply packets correctly as the sender IP of the request is the private IP of the VM. The sender IP of the reply will then also be the IP of the VM and the sending TCP socket will not be able to process the packet as the replier is wrong.

This scenario is not easy to solve.

You would need additional SNAT rules on the Proxmox host that apply when then VM itself tries to send a packet to the public IP masking the sender IP (the VM) with its own IP so that when the packet is received by the VM (via DNAT) the VM thinks the Proxmox host is the sender and sends the answer back there.

 

[dsrvlaga]

The DNAT rules will not be active in this case as the packet will not come in via vmbr0 (from the Internet) but vmbr1 (from the VM).
Even if that would be corrected you will not get the reply packets correctly as the sender IP of the request is the private IP of the VM. The sender IP of the reply will then also be the IP of the VM and the sending TCP socket will not be able to process the packet as the replier is wrong.

This scenario is not easy to solve.

You would need additional SNAT rules on the Proxmox host that apply when then VM itself tries to send a packet to the public IP masking the sender IP (the VM) with its own IP so that when the packet is received by the VM (via DNAT) the VM thinks the Proxmox host is the sender and sends the answer back there.

Okay thank you very much for the fast reply.

Now is my question: What's the best thing we can do (in your opinion)? Is there another setup that avoids this issue or should we try to setup those additional SNAT rules as you suggested?

 

[gurubert]

You could add entries in /etc/hosts of the VM with the names in question and the private IP of the VM. This overrides the global DNS.

 

[dsrvlaga]

You could add entries in /etc/hosts of the VM with the names in question and the private IP of the VM. This overrides the global DNS.

Thanks this works great thank you very much for the help.

Now that this definitely works, I do want to try adding the SNAT as a long-term solution. Do you by chance have a great source where I can find out how to add it?

 

[ICE13]

This solution works for me:

iptables -t nat -A PREROUTING -i vmbr0 -p tcp -d HOST.PUBLIC.IP.ADDRESS -j MARK --set-mark 0x200/0x200
iptables -t nat -A PREROUTING -i vmbr0 -p tcp -d HOST.PUBLIC.IP.ADDRESS -j DNAT --to IP.ADDRESS.OF.VMBR0
iptables -t nat -A POSTROUTING -o vmbr0 -m mark --mark 0x200 -j MASQUERADE