[SOLVED] Masquerade network unable to resolve names (DNS)

[Ouji]

Hi! I've installed Proxmox VE 6 on top of Debian 10 (as my hosting provider does not provide IPMI or console access).

Since I also cannot buy additional IPs, I resorted to use the masquerade network to make all my VMs/LXC connect to the outside world.

I've managed to make it ping any IP from inside the VMs, but I cannot make DNS work, so I can't install any packages or anything else.

Here's my host network config

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

allow-hotplug eth0

iface eth0 inet static
        address  144.xxx.xxx.xxx
        netmask  24
        gateway  144.xxx.xxx.1
        dns-nameservers 8.8.8.8
        dns-search vitor.pw
# dns-* options are implemented by the resolvconf package, if installed

iface eth1 inet manual

auto vmbr2
iface vmbr2 inet static
    address 10.0.0.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

I don't what I'm actually doing wrong and I'd live if someone can help me out.

Also, I've tried setting up the /etc/resolv.conf file with nameserver values, but it didn't work. I've searched extensively on the subject, but as this is my first time, there is probably something that I'm missing.

Thank you!

 

[lhorace]

Can you ping any IP from the hypervisor ?

 

[Ouji]

Can you ping any IP from the hypervisor ?

Yes, the hypervisor is connected to the internet. I think that might be something related to ufw firewall. I just don't know what as I'm kinda noob on that, exactly why I avoid iptables like the plague.

 

[Stoiko Ivanov]

Which host has the dns-problem?
* The PVE-node/hypervisor? (i.e. can you `ping www.google.com` from a shell on the hypervisor)?
** if not either your provider is blocking access to external DNS-servers (in that case contact them for an internal one)
** or you have some kind of iptables rule, which prevents access to the DNS-server - post the output of `iptables-save`

* a guest on the PVE-node - in that case please post the guests config and:
** try to ping the node from the guest `ping 10.0.0.1`
** which DNS-server have you configured on the guest?

I hope this helps!

 

[Ouji]

Which host has the dns-problem?
* The PVE-node/hypervisor? (i.e. can you `ping www.google.com` from a shell on the hypervisor)?
** if not either your provider is blocking access to external DNS-servers (in that case contact them for an internal one)
** or you have some kind of iptables rule, which prevents access to the DNS-server - post the output of `iptables-save`

* a guest on the PVE-node - in that case please post the guests config and:
** try to ping the node from the guest `ping 10.0.0.1`
** which DNS-server have you configured on the guest?

I hope this helps!

The problem was related to UFW, I installed it on the host (as a part of the automation for new servers), so I ended up reinstalling everything and now this problem has been solved. Now I have a few different issues and I hope you can answer my queries, if possible.

Again, my setup is a NAT masquerade as I don't have any spare IPs.
After the reinstall, I configured the network, but as I don't know a lot of iptables, I ended up using PVE Firewall. Using only on the cluster (datacenter) level, protects everything below it? Nodes, VMs, CTs, etc?

Also, I installed a reverse proxy that has a webgui in port 81, so I did as it is in the file to redirect the port. Also, I learned that the last two lines are necessary for the pve-firewall to work with NAT. I opened all 3 ports on the pve-firewall, and accessing everything via my ip:81. ip:80 and ip:443 works, but nginx is unable to work as a reverse proxy somehow. Also tried Caddy and it didn't work as well. Also, if I try the FQDN that I pointed to my ip with the port, i.e. foo.com:81 or foo.com:80 it also doesn't work. So I don't think that there's something wrong with the webservers.

/etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
    address 144.xxx.xxx.xxx/24
    gateway 144.xxx.xxx.xxx
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 8.8.8.8
    

auto vmbr2
iface vmbr2 inet static
    address 192.168.0.254
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0
  
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '192.168.0.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '192.168.0.0/24' -o eth0 -j MASQUERADE

    # 100
    post-up iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 81 -j DNAT --to 192.168.0.100:81
    post-down iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 81 -j DNAT --to 192.168.0.100:81
    post-up iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.0.100:80
    post-down iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 192.168.0.100:80
    post-up iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to 192.168.0.100:443
    post-down iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to 192.168.0.100:443

    # Allow NAT working with the built-in firewall
    post-up   iptables -t raw -I PREROUTING  -i fwbr+ -j CT --zone 1
    post-down iptables -t raw -D PREROUTING  -i fwbr+ -j CT --zone 1

I just wanna make sure that my server is protected without having to configure multiple firewalls. For instance, as soon as I get any of the webservers/reverse proxies, no port will be opened in the node, as all the communication will be made inside the private network.

Thank you.

 

[Stoiko Ivanov]

After the reinstall, I configured the network, but as I don't know a lot of iptables, I ended up using PVE Firewall. Using only on the cluster (datacenter) level, protects everything below it? Nodes, VMs, CTs, etc?

You need to enable the firewall on all levels (you want to protect):
* Datacenter (this is a global on-/off-switch)
* Host (to firewall the PVE-node itself)
* Guest (switch for an individual guest)
* Guests' interface (if you want to exclude certain interfaces)

nginx is unable to work as a reverse proxy somehow. Also tried Caddy and it didn't work as well. Also, if I try the FQDN that I pointed to my ip with the port, i.e. foo.com:81 or foo.com:80 it also doesn't work. So I don't think that there's something wrong with the webservers.

I guess if 2 proxies do not work it's more likely not related to a particular software (i.e. you should be able to stick with 'nginx')
Regarding the FQDN:81 - does DNS-resolution work - i.e. does FQDN resolve to your public-ip?
Does the proxy run on the PVE-node directly or is it in a guest (container/VM)?

In any case I suggest you look at the arriving traffic with tcpdump at various points:
* physical ingress interface
* bridge
* guest tap/fwbr device on the node
* inside the guest

that way you find out where the packets get dropped - mangled incorrectly


I hope this helps!

 

[Ouji]

You need to enable the firewall on all levels (you want to protect):
* Datacenter (this is a global on-/off-switch)
* Host (to firewall the PVE-node itself)
* Guest (switch for an individual guest)
* Guests' interface (if you want to exclude certain interfaces)

That doesn't make a lot of sense to me, to be honest. Mainly because my networking is NAT, so if I have my ports closed on the cluster or on the node, but open in the guests, how come would a request enter in a port that is closed in the node, but it is open in the guest? The node and the guest are listening on the same internet-faced interface.

I guess if 2 proxies do not work it's more likely not related to a particular software (i.e. you should be able to stick with 'nginx')
Regarding the FQDN:81 - does DNS-resolution work - i.e. does FQDN resolve to your public-ip?
Does the proxy run on the PVE-node directly or is it in a guest (container/VM)?

In any case I suggest you look at the arriving traffic with tcpdump at various points:
* physical ingress interface
* bridge
* guest tap/fwbr device on the node
* inside the guest

that way you find out where the packets get dropped - mangled incorrectly


I hope this helps!

The FQDN resolves to my public IP, but for some reason I can't ping it. If I restart my node, I can ping it fine, but as soon as I ping one of the FQDN pointed to my node, it stops pinging overall. I'm honestly very confused by all this.

About the tcpdump I'm gonna have to read on it as I don't know how to use it. Right now I don't have the time, but I will try to play with the firewall more later on to find out if that's the issue.

Thank you again for your reply!

 

[Stoiko Ivanov]

The FQDN resolves to my public IP, but for some reason I can't ping it. If I restart my node, I can ping it fine, but as soon as I ping one of the FQDN pointed to my node, it stops pinging overall. I'm honestly very confused by all this.

This sounds very odd - unless you have some kind of rate-limiting in your firewall - but in any case - I'd rather take a look whether ping can indeed resolve the name (it should write the ip address it tries to ping)
if it's the correct ip-address then take a look at the logs of your node (`journalctl -r`) maybe there is a hint there...

 

[Ouji]

This sounds very odd - unless you have some kind of rate-limiting in your firewall - but in any case - I'd rather take a look whether ping can indeed resolve the name (it should write the ip address it tries to ping)
if it's the correct ip-address then take a look at the logs of your node (`journalctl -r`) maybe there is a hint there...

Well. I did a reinstall in the node and the issue is now solved. Thank you very much!

 

[Stoiko Ivanov]

Glad the issue is resolved! - please mark the thread as 'SOLVED'
Thanks!