I'm having a weird behavior since the migration from the latest 7.3 to 7.4-3
I have a proxmox hosted server (OVH) with a single public IPV4.
I have a single LXC container and on the host a list of NAT and ip forwarding settings so most of the requests (http, https, smtp, imap,...) are natted to the LXC.
Everything was working fine, but since the 7.4 migration the LXC can't access to it's own resources if it tries to access them using the domain name (or public ip) associated with it.
Here is an example :
from the LXC :
ping mywebsite.com
-> is properly resolved to the public ipv4 (so it's not DNS related)
but :
telnet mywebsite.com 443
-> ends with a timeout
on the other end telnet proxmox.com 443 works properly
Of course, everything else works perfectly, for example from any computer in the world I can access to mywebsite.com without any problem (the NAT works fine). The problem occurs for example when I have a schedule task on the LXC that makes a curl or wget call to something like https://mywebsite.com/schedule_task_number_1.php
It also is a problem for a pdf generation service (from html pages) that calls css, fonts, images using the domain name.
I have 2 servers with the same kind of configuration, and both experience the same problem.
The initial configuration is quite basic:
- vmbr0 is linked to the real network interface
- creation of a vmbr1 bridge with a 192.168.50.1/24 IP/CIDR
- the LXC has the 192.168.50.10 IP
- proxmox firewall is set to off at the datacenter level. I've seen that there is an "ebtables" switch, but I'm not sure if I am supposed to play with it. As it is a production cluster, with people working on it, I don't want to make more harm than good
- /proc/sys/net/ipv4/ip_forward is set to 1 (using the /etc/sysctl.conf file so it stays at 1 even after a reboot)
- the main iptables rules (at least the most important IMHO):
sudo iptables -t nat -A POSTROUTING -s '192.168.50.0/24' -o vmbr0 -j MASQUERADE
# allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT ; sudo iptables -A OUTPUT -o lo -j ACCEPT
# allow going outside:
sudo iptables -A OUTPUT -p udp --dport 53 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT
# keep a connected status:
sudo iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT ; sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ; sudo iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# ICMP, ping, ... :
sudo iptables -A INPUT -p icmp -j ACCEPT ; sudo iptables -A OUTPUT -p icmp -j ACCEPT ; sudo iptables -A OUTPUT -p udp --dport 123 -j ACCEPT
# the NAT rules:
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 192.168.50.10:80
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 192.168.50.10:443
... (many more for ssh, smtp, pop, imap, ...)
#finally the drop rules:
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
the rules are kept persistent using "iptables-persistent"
iptables -L and iptables -L -n -t nat
both show exactly what is planned
Some debugging I tried:
- disable the "iptables -P OUTPUT DROP" (so ACCEPT instead). No change.
- play with /proc/sys/net/ipv4/ip_forward as no impact (0 or 1 nothing changes). Is this setting supposed to have an impact on anything nowdays?
- traceroute mywebsite.com gives 30 hops with only * * * * which makes me think that there is something wrong on that level.
- if I manually edit the LXC /etc/hosts and add a line with "127.0.0.1 mywebsite.com" it "solves" most of my problems as the LXC has direct acces to its own resources, but it is not a proper solution as I have many domain and subdomain names and I don't think it should be necessary (it wasn't with the 7.3 version).
- add "iptables -t nat -A PREROUTING -d xx.xx.xx.xx -p tcp --dport 443 -j DNAT --to-destination 192.168.50.10" whith xx.xx.xx.xx being the public IP, to "force" the NAT even if it comes from the inside (not attached to a specific input) -> no change
- check ebtables (-L) : no rules, everything is set to ACCEPT)
What can I try to investigate more and solve this problem?
I have a proxmox hosted server (OVH) with a single public IPV4.
I have a single LXC container and on the host a list of NAT and ip forwarding settings so most of the requests (http, https, smtp, imap,...) are natted to the LXC.
Everything was working fine, but since the 7.4 migration the LXC can't access to it's own resources if it tries to access them using the domain name (or public ip) associated with it.
Here is an example :
from the LXC :
ping mywebsite.com
-> is properly resolved to the public ipv4 (so it's not DNS related)
but :
telnet mywebsite.com 443
-> ends with a timeout
on the other end telnet proxmox.com 443 works properly
Of course, everything else works perfectly, for example from any computer in the world I can access to mywebsite.com without any problem (the NAT works fine). The problem occurs for example when I have a schedule task on the LXC that makes a curl or wget call to something like https://mywebsite.com/schedule_task_number_1.php
It also is a problem for a pdf generation service (from html pages) that calls css, fonts, images using the domain name.
I have 2 servers with the same kind of configuration, and both experience the same problem.
The initial configuration is quite basic:
- vmbr0 is linked to the real network interface
- creation of a vmbr1 bridge with a 192.168.50.1/24 IP/CIDR
- the LXC has the 192.168.50.10 IP
- proxmox firewall is set to off at the datacenter level. I've seen that there is an "ebtables" switch, but I'm not sure if I am supposed to play with it. As it is a production cluster, with people working on it, I don't want to make more harm than good
- /proc/sys/net/ipv4/ip_forward is set to 1 (using the /etc/sysctl.conf file so it stays at 1 even after a reboot)
- the main iptables rules (at least the most important IMHO):
sudo iptables -t nat -A POSTROUTING -s '192.168.50.0/24' -o vmbr0 -j MASQUERADE
# allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT ; sudo iptables -A OUTPUT -o lo -j ACCEPT
# allow going outside:
sudo iptables -A OUTPUT -p udp --dport 53 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT ; sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT
# keep a connected status:
sudo iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT ; sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ; sudo iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# ICMP, ping, ... :
sudo iptables -A INPUT -p icmp -j ACCEPT ; sudo iptables -A OUTPUT -p icmp -j ACCEPT ; sudo iptables -A OUTPUT -p udp --dport 123 -j ACCEPT
# the NAT rules:
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 80 -j DNAT --to 192.168.50.10:80
iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 443 -j DNAT --to 192.168.50.10:443
... (many more for ssh, smtp, pop, imap, ...)
#finally the drop rules:
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
the rules are kept persistent using "iptables-persistent"
iptables -L and iptables -L -n -t nat
both show exactly what is planned
Some debugging I tried:
- disable the "iptables -P OUTPUT DROP" (so ACCEPT instead). No change.
- play with /proc/sys/net/ipv4/ip_forward as no impact (0 or 1 nothing changes). Is this setting supposed to have an impact on anything nowdays?
- traceroute mywebsite.com gives 30 hops with only * * * * which makes me think that there is something wrong on that level.
- if I manually edit the LXC /etc/hosts and add a line with "127.0.0.1 mywebsite.com" it "solves" most of my problems as the LXC has direct acces to its own resources, but it is not a proper solution as I have many domain and subdomain names and I don't think it should be necessary (it wasn't with the 7.3 version).
- add "iptables -t nat -A PREROUTING -d xx.xx.xx.xx -p tcp --dport 443 -j DNAT --to-destination 192.168.50.10" whith xx.xx.xx.xx being the public IP, to "force" the NAT even if it comes from the inside (not attached to a specific input) -> no change
- check ebtables (-L) : no rules, everything is set to ACCEPT)
What can I try to investigate more and solve this problem?
Last edited: