VM can not access internet - Private IP

be_tnt · Nov 19, 2018

Hello!

Let me try to explain what I am trying to do. On the proxmox server, I will have 4 VMs (centOS 7):

a load balancer / reverse proxy
2 web servers
1 database servers

The load balancer will have a public IP but the 3 other ones will have a private IP.

On the host, I do have the following interfaces :

Code:

auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual

# public IP
auto vmbr0
iface vmbr0 inet static
    address  XX.XX.XX.XX
    netmask  255.255.255.0
    gateway  XX.XX.XX.254
    bridge-ports eno3
    bridge-stp off
    bridge-fd 0

# private IP
auto vmbr1
iface vmbr1 inet static
    address  10.0.1.1
    netmask  255.255.255.0
    bridge-ports none
    bridge-stp off
    bridge-fd 0

    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.0.1.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.1.0/24' -o vmbr0 -j MASQUERADE

I have installed one of the webserver with the IP 10.0.1.10. It can ping the host and the host can ping it.
I can do ssh to the VM from the host but it's really slow.

The VM can not ping google.be.

Following https://cyberpersons.com/2016/07/27/setup-nat-proxmox/, I set up an iptable rule on the host:

Code:

iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 3033 -j DNAT --to 10.0.1.10:22

But when trying to access VM ssh, it does not work.
Command used:

Code:

# ssh -p 3033 root@<proxmox server IP>

Received response: Operation timed out

Firewall are enabled on each level. Port 3033 is allowed on host firewall and ssh port is allowed on VM port.

Below the network config for the VM :

Any help would be appreciated. Thx!

wolfgang · Nov 20, 2018

Hi,

please send the output of
You can mask the public IP addresses

Code:

iptalbes -L
ip a
cat /proc/sys/net/ipv4/ip_forward
qm config <VMID>

be_tnt · Nov 20, 2018

Hello Wolfgang!

Thx for trying to help. Below the requested information:

ip a

Code:

root@dobby:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:c2:a4:56 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:c2:a4:57 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 0c:c4:7a:c2:a5:86 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:c2:a5:87 brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:c2:a5:86 brd ff:ff:ff:ff:ff:ff
    inet XX.XX.XX.XX/24 brd XX.XX.XX255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 2001:41d0:203:1849::/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fec2:a586/64 scope link
       valid_lft forever preferred_lft forever
7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:45:4d:49:01:e8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::ace2:34ff:fe84:dc7f/64 scope link
       valid_lft forever preferred_lft forever
8: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether d6:83:13:ac:a1:44 brd ff:ff:ff:ff:ff:ff
9: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:9f:4d:fb:5e:cd brd ff:ff:ff:ff:ff:ff
10: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether f6:45:4d:49:01:e8 brd ff:ff:ff:ff:ff:ff
11: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether b2:9f:4d:fb:5e:cd brd ff:ff:ff:ff:ff:ff

The file /proc/sys/net/ipv4/ip_forward contains 1.

Command qm config <VMID>:

Code:

root@dobby:~# qm config 100
bootdisk: scsi0
cores: 4
ide2: local:iso/CentOS-7-x86_64-DVD-1804.iso,media=cdrom
keyboard: fr-be
memory: 8192
name: minerva
net0: virtio=C6:89:2A:9B:14:52,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,size=250G
scsihw: virtio-scsi-pci
smbios1: uuid=b54afec5-9c32-47ef-809a-7f5fdaee81f1
sockets: 1
vmgenid: 0a0f04cf-749b-4ae7-9329-18f372672ae1

Command iptables -L (quite long):

Code:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
f2b-proxmox  tcp  --  anywhere             anywhere             multiport dports https,http,8006
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh
PVEFW-INPUT  all  --  anywhere             anywhere           

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
PVEFW-FORWARD  all  --  anywhere             anywhere           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
PVEFW-OUTPUT  all  --  anywhere             anywhere           

Chain PVEFW-Drop (2 references)
target     prot opt source               destination         
PVEFW-reject  tcp  --  anywhere             anywhere             tcp dpt:whois
PVEFW-DropBroadcast  all  --  anywhere             anywhere           
ACCEPT     icmp --  anywhere             anywhere             icmp fragmentation-needed
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
DROP       all  --  anywhere             anywhere             ctstate INVALID
DROP       udp  --  anywhere             anywhere             multiport dports loc-srv,microsoft-ds
DROP       udp  --  anywhere             anywhere             udp dpts:netbios-ns:netbios-ssn
DROP       udp  --  anywhere             anywhere             udp spt:netbios-ns dpts:1024:65535
DROP       tcp  --  anywhere             anywhere             multiport dports loc-srv,netbios-ssn,microsoft-ds
DROP       udp  --  anywhere             anywhere             udp dpt:1900
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
DROP       udp  --  anywhere             anywhere             udp spt:domain
           all  --  anywhere             anywhere             /* PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ */

Chain PVEFW-DropBroadcast (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type ANYCAST
DROP       all  --  anywhere             base-address.mcast.net/4
           all  --  anywhere             anywhere             /* PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w */

Chain PVEFW-FORWARD (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
PVEFW-FWBR-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-in fwln+ --physdev-is-bridged
PVEFW-FWBR-OUT  all  --  anywhere             anywhere             PHYSDEV match --physdev-out fwln+ --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw */

Chain PVEFW-FWBR-IN (1 references)
target     prot opt source               destination         
PVEFW-smurfs  all  --  anywhere             anywhere             ctstate INVALID,NEW
tap100i0-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap100i0 --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:/naDZxJ06t8Dx9DQtmus9NvdHEA */

Chain PVEFW-FWBR-OUT (1 references)
target     prot opt source               destination         
tap100i0-OUT  all  --  anywhere             anywhere             PHYSDEV match --physdev-in tap100i0 --physdev-is-bridged
           all  --  anywhere             anywhere             /* PVESIG:wA3mj3VIKyC/rlY95PCFN7paR5s */

Chain PVEFW-HOST-IN (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere           
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
PVEFW-smurfs  all  --  anywhere             anywhere             ctstate INVALID,NEW
RETURN     igmp --  anywhere             anywhere           
RETURN     tcp  --  anywhere             anywhere             tcp dpts:5900:5999
RETURN     tcp  --  anywhere             anywhere             tcp dpt:8006
RETURN     icmp --  anywhere             anywhere             icmp echo-request
RETURN     udp  --  anywhere             anywhere             udp dpt:ntp
RETURN     tcp  --  anywhere             anywhere             tcp dpt:http
RETURN     tcp  --  anywhere             anywhere             tcp dpt:ssh
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:8006
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpts:5900:5999
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:3128
RETURN     tcp  --  anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:ssh
RETURN     udp  --  xx.xx.xx.0/24       xx.xx.xx.0/24       udp dpts:5404:5405
RETURN     udp  --  xx.xx.xx.0/24       anywhere             ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405
PVEFW-Drop  all  --  anywhere             anywhere           
NFLOG      all  --  anywhere             anywhere             nflog-prefix  ":0:1:PVEFW-HOST-IN: policy DROP: "
DROP       all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:caQnOQIumkbqNPfsPfondzVQBdQ */

Chain PVEFW-HOST-OUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere           
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
RETURN     igmp --  anywhere             anywhere           
RETURN     tcp  --  anywhere             xx.xx.xx..0/24       tcp dpt:8006
RETURN     tcp  --  anywhere             xx.xx.xx.0/24       tcp dpt:ssh
RETURN     tcp  --  anywhere             xx.xx.xx.0/24       tcp dpts:5900:5999
RETURN     tcp  --  anywhere             xx.xx.xx.0/24       tcp dpt:3128
RETURN     udp  --  anywhere             xx.xx.xx.0/24       udp dpts:5404:5405
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405
RETURN     all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:3yAcVpr0ewHg8DxFR7g9C8B8xxE */

Chain PVEFW-INPUT (1 references)
target     prot opt source               destination         
PVEFW-HOST-IN  all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk */

Chain PVEFW-OUTPUT (1 references)
target     prot opt source               destination         
PVEFW-HOST-OUT  all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0 */

Chain PVEFW-Reject (0 references)
target     prot opt source               destination         
PVEFW-reject  tcp  --  anywhere             anywhere             tcp dpt:whois
PVEFW-DropBroadcast  all  --  anywhere             anywhere           
ACCEPT     icmp --  anywhere             anywhere             icmp fragmentation-needed
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
DROP       all  --  anywhere             anywhere             ctstate INVALID
PVEFW-reject  udp  --  anywhere             anywhere             multiport dports loc-srv,microsoft-ds
PVEFW-reject  udp  --  anywhere             anywhere             udp dpts:netbios-ns:netbios-ssn
PVEFW-reject  udp  --  anywhere             anywhere             udp spt:netbios-ns dpts:1024:65535
PVEFW-reject  tcp  --  anywhere             anywhere             multiport dports loc-srv,netbios-ssn,microsoft-ds
DROP       udp  --  anywhere             anywhere             udp dpt:1900
DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
DROP       udp  --  anywhere             anywhere             udp spt:domain
           all  --  anywhere             anywhere             /* PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo */

Chain PVEFW-SET-ACCEPT-MARK (2 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             MARK or 0x80000000
           all  --  anywhere             anywhere             /* PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY */

Chain PVEFW-logflags (5 references)
target     prot opt source               destination         
NFLOG      all  --  anywhere             anywhere             nflog-prefix  ":0:1:PVEFW-logflags: DROP: "
DROP       all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:B9JfHztz+i9ARD3xG6Edm8T3MvM */

Chain PVEFW-reject (6 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
DROP       all  --  base-address.mcast.net/4  anywhere           
DROP       icmp --  anywhere             anywhere           
REJECT     tcp  --  anywhere             anywhere             reject-with tcp-reset
REJECT     udp  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     icmp --  anywhere             anywhere             reject-with icmp-host-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
           all  --  anywhere             anywhere             /* PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc */

Chain PVEFW-smurflog (2 references)
target     prot opt source               destination         
NFLOG      all  --  anywhere             anywhere             nflog-prefix  ":0:1:PVEFW-smurflog: DROP: "
DROP       all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:U0bhVR22AI+578udCidFZuCZn7s */

Chain PVEFW-smurfs (2 references)
target     prot opt source               destination         
RETURN     all  --  default              anywhere           
PVEFW-smurflog  all  --  anywhere             anywhere            [goto]  ADDRTYPE match src-type BROADCAST
PVEFW-smurflog  all  --  base-address.mcast.net/4  anywhere            [goto]
           all  --  anywhere             anywhere             /* PVESIG:HssVe5QCBXd5mc9kC88749+7fag */

Chain PVEFW-tcpflags (0 references)
target     prot opt source               destination         
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:SYN,RST/SYN,RST
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp flags:FIN,SYN/FIN,SYN
PVEFW-logflags  tcp  --  anywhere             anywhere            [goto]  tcp spt:0 flags:FIN,SYN,RST,ACK/SYN
           all  --  anywhere             anywhere             /* PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo */

Chain f2b-proxmox (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere           

Chain f2b-sshd (1 references)
target     prot opt source               destination         
REJECT     all  --  220.red-88-3-231.dynamicip.rima-tde.net  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  88.144.55.13         anywhere             reject-with icmp-port-unreachable
REJECT     all  --  na-201-156-122-7.static.avantel.net.mx  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  mail46.gtdinternet.com  anywhere             reject-with icmp-port-unreachable
REJECT     all  --  188.166.255.174      anywhere             reject-with icmp-port-unreachable
REJECT     all  --  185.254.120.6        anywhere             reject-with icmp-port-unreachable
REJECT     all  --  180.76.155.59        anywhere             reject-with icmp-port-unreachable
REJECT     all  --  167.99.77.63         anywhere             reject-with icmp-port-unreachable
REJECT     all  --  abts-kk-dynamic-194.172.167.122.airtelbroadband.in  anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere           

Chain tap100i0-IN (1 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
PVEFW-Drop  all  --  anywhere             anywhere           
NFLOG      all  --  anywhere             anywhere             nflog-prefix  ":100:3:tap100i0-IN: policy DROP: "
DROP       all  --  anywhere             anywhere           
           all  --  anywhere             anywhere             /* PVESIG:OjG3yy8rS+mF58lP7OOIlzITKN4 */

Chain tap100i0-OUT (1 references)
target     prot opt source               destination         
PVEFW-SET-ACCEPT-MARK  udp  --  anywhere             anywhere            [goto]  udp spt:bootpc dpt:bootps
DROP       all  --  anywhere             anywhere             MAC ! C6:89:2A:9B:14:52
MARK       all  --  anywhere             anywhere             MARK and 0x7fffffff
PVEFW-SET-ACCEPT-MARK  all  --  anywhere             anywhere            [goto]
           all  --  anywhere             anywhere             /* PVESIG:j+bYiBsPORQKU/flDmr0DuibHFs */

And maybe this one could be usefull too: iptables -L -t nat

Code:

root@dobby:/tmp# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  10.0.1.0/24          anywhere

Thx!

be_tnt · Nov 21, 2018

I have disabled the firewall on datacenter and on vm and now the VM can access internet. So it seems it's missing something in the firewall rules.

be_tnt · Nov 21, 2018

When I just reactive the firewall on datacenter level, it drops connection to VM (SSH NAT through the host) and VM does not access the internet anymore.

The settings for the datacenter firewall is:

And:

Any idea on what could be missing?

Thx!

be_tnt · Nov 21, 2018

Progressing .... I have noticed that on network device (net0) on my guest (menu "Hardware"), the firewall option was checked. I removed it and now with the above datacenter firewall settings, my guest can access internet. And I can connect to my guest with ssh thanks to NAT.

Honestly it's quite difficult to understand how the firewall is working. Is there a documentation for "dummy" ? I read the proxmox documentation about firewall and I did not see anything about this option.

haxus · Feb 15, 2021

Thank be_tnt for your investigation. Your workaround, having disabled firewall in hardware (networkcard) section of the vm gives me at least the possibillity to reach the internet. However, It would be awesome if someone coulöd explain why NAT stops working if firewall is enabled and how to get firewall plus nat propertly to work.

Thake you

spirit · Feb 15, 2021

you need to add

Code:

post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-up   iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

to have the firewall working with nat

haxus · Mar 2, 2021

Thanks -- works!!!

Search

Search

VM can not access internet - Private IP

be_tnt

Member

wolfgang

Proxmox Retired Staff

be_tnt

Member

be_tnt

Member

be_tnt

Member

be_tnt

Member

haxus

New Member

spirit

Distinguished Member

haxus

New Member