Proxmox VM firewall not working on routed hetzner config.

U

Undergrid

Well-Known Member
Jul 15, 2018
31
3
48
49
I have a Hetzner dedicated server running proxmox with a IPv4 /29 and IPv6 /64 routed to the virtual machines. Until recently I have left the hetzner firewall enabled along with the proxmox one, but I discovered some issues with it and a pfSense vm I wanted to run, so I turned it off... Only to discover the proxmox firewall doesn't seem to have any effect on the VM's.

Each VM has the "firewall=1" option on the network device in the hardware configuration and "Firewall" set to Yes under VM > Firewall > Options, with a DROP input policy. There is no rule allowing access to SSH, and yet an nmap scan of the VMs show SSH available externally.

A (ip address redacted) network config of my server is below. Is the firewall supposed to work with this configuration?

Code: 
source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
  address 95.217.AAA.BBB
  netmask 255.255.255.192
  gateway 95.217.AAA.CCC
  # route 95.217.34.128/26 via 95.217.34.129
  up route add -net 95.217.AAA.BBB netmask 255.255.255.192 gw 95.217.AAA.CCC dev enp0s31f6

iface enp0s31f6 inet6 static
  address 2a01:4f9:XXXX:XXXX::2
  netmask 128
  gateway fe80::1

auto vmbr0
iface vmbr0 inet static
  address 95.217.AAA.BBB
  netmask 255.255.255.192
  bridge-ports none
  bridge-stp off
  bridge-fd 0
  bridge_maxwait 0
  up route add -net 95.216.DDD.EEE/29 dev vmbr0

iface vmbr0 inet6 static
  address 2a01:4f9:XXXX:XXXX::2
  netmask 64

auto vmbr1
iface vmbr1 inet static
  address 192.168.100.1
  netmask 255.255.0.0
  bridge-ports none
  bridge-stp off
  bridge-fd 0
  bridge_maxwait 0
 
Hi,
is the firewall enabled at Datacenter level? What is the output of `iptables-save`?
Also note that ssh communication is allowed by default for the cluster network (although to the nodes, not the VMs).
 
Hi Chris,

The Datacenter level has "Firewall" set to Yes under Firewall > Options. The firewall does appear to be protecting the proxmox host though (which is not clustered) as I can (for example) toggle on a HTTP rule and see port 80 go from filtered to closed on the nmap scan.

I've also tried turning on the firewall between a VM and an LXC both on vmbr1 and that seems to have no effect either, I can still ping and use HTTP traffic between them without restriction, even when the firewalls have no ACCEPT rules (actually, no rules at all).

Output of iptables-save:
Code: 
# Generated by iptables-save v1.6.0 on Thu May 16 17:27:41 2019
*filter
:INPUT ACCEPT [8:416]
:FORWARD ACCEPT [18:764]
:OUTPUT ACCEPT [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
:tap100i0-IN - [0:0]
:tap100i0-OUT - [0:0]
:tap100i1-IN - [0:0]
:tap100i1-OUT - [0:0]
:tap900i0-IN - [0:0]
:tap900i0-OUT - [0:0]
:veth101i0-IN - [0:0]
:veth101i0-OUT - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m physdev --physdev-out tap100i0 --physdev-is-bridged -j tap100i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap100i1 --physdev-is-bridged -j tap100i1-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out tap900i0 --physdev-is-bridged -j tap900i0-IN
-A PVEFW-FWBR-IN -m physdev --physdev-out veth101i0 --physdev-is-bridged -j veth101i0-IN
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:VENEOXYpaELX0Wq0MGuv4v3z0SA"
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap100i0 --physdev-is-bridged -j tap100i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap100i1 --physdev-is-bridged -j tap100i1-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in tap900i0 --physdev-is-bridged -j tap900i0-OUT
-A PVEFW-FWBR-OUT -m physdev --physdev-in veth101i0 --physdev-is-bridged -j veth101i0-OUT
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:QKW8dnAxmixtx6Baeb47hdmoba8"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p icmp -m icmp --icmp-type 8 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:h9eU8MC4L3rvSshjpSZx0ObGvvA"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:0zVqGNg5V7YiSxt0h+sEYJsXD+M"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
-A tap100i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap100i0-IN -j PVEFW-Drop
-A tap100i0-IN -j DROP
-A tap100i0-IN -m comment --comment "PVESIG:dyBT7vHzkJ/DuuTmaQj+ta/b9fk"
-A tap100i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap100i0-OUT -m mac ! --mac-source 52:C6:A3:2A:B1:D6 -j DROP
-A tap100i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap100i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap100i0-OUT -m comment --comment "PVESIG:YueyCFOkGMj7Yv5EqGfuKPBiGZE"
-A tap100i1-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap100i1-IN -j PVEFW-Drop
-A tap100i1-IN -j DROP
-A tap100i1-IN -m comment --comment "PVESIG:B0QUyLtkBzqkactQx1JAzuwFda8"
-A tap100i1-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap100i1-OUT -m mac ! --mac-source 5E:31:91:47:51:AC -j DROP
-A tap100i1-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap100i1-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap100i1-OUT -m comment --comment "PVESIG:rcp+vze2iisD+z3A8OLKaz1JSms"
-A tap900i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A tap900i0-IN -p udp -m udp --dport 1194 -j ACCEPT
-A tap900i0-IN -p tcp -m set --match-set PVEFW-0-home-v4 src -m tcp --dport 8006 -j ACCEPT
-A tap900i0-IN -p tcp -m set --match-set PVEFW-0-work-v4 src -m tcp --dport 8006 -j ACCEPT
-A tap900i0-IN -p tcp -m tcp --dport 443 -j ACCEPT
-A tap900i0-IN -p tcp -m tcp --dport 80 -j ACCEPT
-A tap900i0-IN -j PVEFW-Drop
-A tap900i0-IN -j DROP
-A tap900i0-IN -m comment --comment "PVESIG:4C9RyRQAq7oTqv8AYTztvD/u+PE"
-A tap900i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A tap900i0-OUT -m mac ! --mac-source EE:C5:94:1D:69:7C -j DROP
-A tap900i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A tap900i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A tap900i0-OUT -m comment --comment "PVESIG:m6Z1sY9YSrcl5FSBoV+BqDt6Dng"
-A veth101i0-IN -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A veth101i0-IN -j PVEFW-Drop
-A veth101i0-IN -j DROP
-A veth101i0-IN -m comment --comment "PVESIG:+7u6nS32qpBm+judbr7fB/UqGvc"
-A veth101i0-OUT -p udp -m udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
-A veth101i0-OUT -m mac ! --mac-source 8A:20:8F:64:20:BA -j DROP
-A veth101i0-OUT -j MARK --set-xmark 0x0/0x80000000
-A veth101i0-OUT -g PVEFW-SET-ACCEPT-MARK
-A veth101i0-OUT -m comment --comment "PVESIG:NfO/88XOl4AhVkfPPNkwPmcRSao"
COMMIT
# Completed on Thu May 16 17:27:41 2019
 
According to your firewall rules incoming traffic should be droped for VM 100 and CT 101 (except DHCP).
You can try to turn on logging to see what's going on https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html#_logging_of_firewall_rules?
Dropped packets should be logged and for incoming connections you should get the first packet before a connection is marked as established.
What version of PVE are you running?
What is the output of `pve-firewall status`?
 
I'm running 5.4-5 (at least that's what it says in the top left of the GUI), and the status output is:

Code: 
root@dianna ~ # pve-firewall status
Status: enabled/running
root@dianna ~ #

I turned on debug level logging for both input and output on VM100 and accessed via https the caddy server running on the VM (which, incidentally, proxies to a service on CT101). Afterwards the log window just says "no content".

It seems that something is causing the firewall to be bypassed and I'm wondering if setting the ip forwarding to make the routed IP addresses work is the cause... Hetzner's instructions set these:

Code: 
net.ipv4.ip_forward=1
net.ipv4.conf.enp0s31f6.send_redirects=0
net.ipv6.conf.all.forwarding=1

But I have another Hetzner box using bridged networking for IPv4 and forwarding for IPv6 (only net.ipv6.conf.all.forwarding=1 is set in that case) and the firewall on that box protects both the IPv4 and IPv6 addresses.
 
This is strange. The best way is probably to debug this with tcpdump on the host as well as the VM/CT to see what is going on.
 
So... I rebooted the host, because when you've tried pretty much everything else, why not? And the firewall has started working for VM's and CT's. I even tried creating a new CT in case it was an issue with VM's/CT's being created since the last reboot, and the firewall works for them too.

I'm at a loss to explain it.

Edit: Meant to say: Chris, thank you for all your help.
 
Undergrid said:
So... I rebooted the host, because when you've tried pretty much everything else, why not? And the firewall has started working for VM's and CT's. I even tried creating a new CT in case it was an issue with VM's/CT's being created since the last reboot, and the firewall works for them too.

I'm at a loss to explain it.

Edit: Meant to say: Chris, thank you for all your help.
Click to expand...
To bad we were not able to further pinpoint the issue, but I'm glad the reboot fixed it for you!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom