[SOLVED] PVE-Firewall doesn't have any effect

S

Schnuffle

Member
Proxmox Subscriber
Oct 18, 2016
15
3
23
54
Hi guys,
I'm trying to setup some firewall rules to protect a VM, but I fail badly and don't have a starting point to find out why.
  • The proxmox host has a public IP X.Y.Z.80 from the network X.Y.Z.64/26
  • The VM uses a bridged network and also has a public IP X.Y.Z.69
I already tried setting the rules on Cluster level, host level or VM level all with no effect. I think I'm missing something.
I read https://pve.proxmox.com/wiki/Firewall and according to that document, I should be fine defining all rules on the cluster/datacenter level.

  • Can a VM rule be used at the cluster/datacenter level?
  • Can somebody show me a cluster rule setup, that prevents access to some VM services ( like SSH or HTTP/S )?
  • Shouldn't the Rule "IN DROP -dest sdm # Drop ALL to SDM server" disallow access to the VMs SSH ( in this case all ports ) port?

Here's some more info about my setup:

pve-firewall localnet
local hostname: virt2
local IP address: X.Y.Z.80
network auto detect: X.Y.Z.64/26
using detected local_network: X.Y.Z.64/26​

Proxmox host network setup
auto lo
iface lo inet loopback
iface lo inet6 loopback

iface enp0s31f6 inet manual

auto vmbr0
iface vmbr0 inet static
address X.Y.Z.80
netmask 255.255.255.192
gateway X.Y.Z.65
bridge_ports enp0s31f6
bridge_stp off
bridge_fd 1
bridge_hello 2
bridge_maxage 12​


Clusterfirewall

[OPTIONS]
enable: 1

[ALIASES]
virt2 X.Y.Z.80 # Virt2 Proxmox host
sdm X.Y.Z.69 # SDM Server


[RULES]
IN ACCEPT -dest virt2 -p tcp -dport 22 # Allow SSH to Virt2 server
IN ACCEPT -dest virt2 -p tcp -dport 3128 # Allow Spice connection
IN ACCEPT -dest virt2 -p tcp -dport 8006 # Allow access to Proxmox web panel
|IN ACCEPT -dest sdm -p tcp -dport 22 # Allow SSH to SDM server
|IN ACCEPT -dest sdm -p tcp -dport 80 # Allow HTTP to SDM server
|IN ACCEPT -dest sdm -p tcp -dport 443 # Allow HTTPS to SDM server
IN DROP -dest sdm # Drop ALL toSDM server

[group ssh] # Allow SSH
IN ACCEPT -p tcp -dport 22 # Allow SSH

[group webserver] # Allo HTTP/S traffic
IN ACCEPT -p tcp -dport 443 # Allow HTTPS
IN ACCEPT -p tcp -dport 80 # Allow HTTP​


Any hints or suggestions welcome.

Regards

Schnuffle
 
Schnuffle said:
I already tried setting the rules on Cluster level, host level or VM level all with no effect.
Click to expand...
For VMs only the VM level is relevant. And you need to make sure the firewall is enabled in all places David mentioned above.
 
Thanxs for the answers, though I'm still stuck.

> Set individual VM network interfaces to firewall
What does that mean?
The interface has a regex pattern 'net\d+', my VM has tap100i0 which of course doesn't fit.
Should it be the "net0" definition out of the VM definition?

I tried to minimize firewall config to do testing:

Proxmox host: X.Y.Z.80/26 ( virt2 )
VM: X.Y.Z.69 ( sdm, VM-ID: 100 )

/etc/pve/firewall/cluster.fw
Code: 
[OPTIONS]
enable: 1
[RULES]
IN ACCEPT -dest X.Y.Z.80 -p tcp -dport 22 # Allow SSH to Virt2 server
IN ACCEPT -dest X.Y.Z.80 -p tcp -dport 3128 # Allow Spice connection
IN ACCEPT -dest X.Y.Z.80 -p tcp -dport 8006 # Allow access to Proxmox web panel

/etc/pve/nodes/virt2/host.fw
Code: 
[OPTIONS]
enable: 1
log_level_in: info

/etc/pve/firewall/100.fw
Code: 
[OPTIONS]
enable: 1
[RULES]

From my understanding that should mean I can't connect to my VM as I've a default policy of "DROP" on the cluster level and on the VM level

To be sure that the rules on VM level are active I stopped/started the VM.

The result is, I can connect to all services on the VM.

I attached a pvereport.txt. You'll see that this server doesn't have a subscription. It's for testing as I don't want to do it on the other 7 servers that have subscriptions and are productive.

As a second test I now added VM rules to the game and restarted the VM:
Code: 
[OPTIONS][/COLOR][/COLOR]
[COLOR=#ff0000][COLOR=#000000]
log_level_in: info
enable: 1

[RULES]

IN DROP -dest 94.130.205.69 -p tcp -dport 3389 # DROP RDP port based
IN DROP -i net0 -dest X.Y.Z.69 -p tcp # Drop all TCP connections
IN RDP(DROP) -i net0 -dest X.Y.Z.69 # DROP RDP

That leads to following iptables-save:

Code: 
# Generated by iptables-save v1.6.0 on Tue Sep 25 13:29:58 2018[/COLOR][/COLOR]
[COLOR=#ff0000][COLOR=#000000]*filter
:INPUT ACCEPT [596:35664]
:FORWARD ACCEPT [2112:108033]
:OUTPUT ACCEPT [6:420]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -d X.Y.Z.80/32 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -d X.Y.Z.80/32 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -d X.Y.Z.80/32 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -s X.Y.Z.64/26 -d X.Y.Z.64/26 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s X.Y.Z.64/26 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j NFLOG --nflog-prefix  ":0:6:PVEFW-HOST-IN: policy DROP: "
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:PTWKc+xIizv5mHxofdlkNRm9PCs"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d X.Y.Z.64/26 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d X.Y.Z.64/26 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d X.Y.Z.64/26 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d X.Y.Z.64/26 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -d X.Y.Z.64/26 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:4abw+i9ZiUSrF8P8ql16RNg6CjI"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Tue Sep 25 13:29:58 2018

I don't see any occurence of my 3 rules.

The more I do tests the less I seem to understand how I should handle the firewall.

When I disable the whole pve firewall and do it manually, I don't have any problems to achieve my goal.


I hope somebody can help me to find the fault in my reasoning

Regards

Schnuffle









 

Attachments

  • pvereport.txt
    26.6 KB · Views: 4
  • Like
Reactions: Ciprian Tomoiaga
As a follow up. Disabling the pve-firewall and setting up the following rules does the job:

Code: 
iptables -F
iptables -A FORWARD -i vmbr0 -d X.Y.Z.69/32 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT # Allow established and related packets
iptables -A FORWARD -i vmbr0 -p tcp -s t.u.v.w/32 -d X.Y.Z.69/32 --dport 22 -j ACCEPT       # Allow SSH from  office
iptables -A FORWARD -i vmbr0 -p tcp -s t2.u2.v2.w2/32 -d X.Y.Z.69/32 --dport 22 -j ACCEPT      # Allow SSH from backup server
iptables -A FORWARD -i vmbr0 -p tcp -s t.u.v.w/32 -d X.Y.Z.69/32 --dport 3389 -j ACCEPT     # Allow RDP from  office
iptables -A FORWARD -i vmbr0 -p tcp -d X.Y.Z.69/32 --dport 80 -j ACCEPT                          # Allow World HTTP
iptables -A FORWARD -i vmbr0 -p tcp -d X.Y.Z.69/32 --dport 443 -j ACCEPT                         # Allow World HTTPS
iptables -A FORWARD -i vmbr0 -d X.Y.Z.69/32   -j DROP                                            # DROP all to sdm
 
Schnuffle said:
> Set individual VM network interfaces to firewall
What does that mean?
Click to expand...

In the VM's Hardware tab, if you open the edit window for a network device, there's a Firewall checkbox you need to check.
You'll then see iptables chains named `tap100i0-IN` or `veth321i2-OUT` along with additional firewall link/bridges.
 
Apologies about not seeing your message earlier. As wbumiller said, you most probably haven't enabled firewalling on the VMs network interface itself. Herewith the relevant entry directly from the VM config file:
Code: 
[root@kvm1a ~]# grep firewall /etc/pve/nodes/kvm1a/qemu-server/*
/etc/pve/nodes/kvm1a/qemu-server/100.conf:net0: virtio=CE:3D:AE:58:AB:A8,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/101.conf:net0: virtio=42:FD:B8:39:2A:5F,bridge=vmbr0,firewall=1,tag=13
/etc/pve/nodes/kvm1a/qemu-server/102.conf:net0: virtio=8E:53:ED:28:25:E3,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/103.conf:net0: virtio=D6:77:82:92:94:E8,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/104.conf:net0: virtio=D6:9C:77:45:7C:18,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/106.conf:net0: virtio=36:92:C7:66:ED:C0,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/107.conf:net0: virtio=BE:2B:63:A7:96:59,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/108.conf:net0: virtio=7A:52:CC:BA:75:BF,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/109.conf:net0: virtio=4E:1D:61:C3:4B:84,bridge=vmbr0,firewall=1,tag=12
/etc/pve/nodes/kvm1a/qemu-server/112.conf:net0: virtio=B2:22:40:E4:DA:5F,bridge=vmbr0,firewall=1,tag=12

Herewith the setting from the UI:
vm_settings.jpg​
 
Last edited:
  • Like
Reactions: Ciprian Tomoiaga and netdesignate
Thanxs guys for the quick help.

In the end you where right, I missed the checkbox in the hardware settings.
I did some more tests now and it behaves as expected.

Again thanxs a lot for the help

To document for others:
  1. Enable Firewall on datacenter level
  2. Enable Firewall on host level
  3. Enable Firewall on VM level
  4. Enable firewall in VM hardware tab on the network card

Regards

Schnuffle
 
  • Like
Reactions: Ciprian Tomoiaga and netdesignate
Spirit, perhaps you or someone else at Proxmox could edit the Wiki and documentation to provide a quick overview of the 4 levels of activation necessary to get this working?

I personally also initially had problems understanding the multiple activation requirements and value the granular flexibility they provide.

PS: Apologies, I see you're a paying subscriber not a Proxmox staff member... ;)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back