Help with PVE firewall configuration

[luison]

Hi, we've never used pve-firewall and now trying to consider it for a server.
This is an OVH installation so we have a host IP and additional ones assigned to a KVM.
I did some testing but none of the rules applied to the KVM itself seemed to work. At the host level they worked but only after a restart.
So my questions, to confirm we can use it or not are:
- can I have it activated at a KVM or CT level only? By that I mean, firewall running without rules on the host.
- in this case would the fact of having an iptables ruling firewall at the host level (or KVM) affect or make it unusable?
- should the rules be active straight away after inserting them via web UI or does some service/network need to be restarted?
- are there any tools/commands to... say update dynamic IPs via a cron from the command lines?

Thanks for any help.

 

[shanreich]

- can I have it activated at a KVM or CT level only? By that I mean, firewall running without rules on the host.

yes, the rules are still on the host, but would only affect VM traffic

- in this case would the fact of having an iptables ruling firewall at the host level (or KVM) affect or make it unusable?

inside the VM there should be no issue - if you want additional rules on the host then you would need to be a bit careful with how you place your chains - depends on your use case then. but pve-firewall shouldn't touch custom chains as long as you do not use the proxmox naming scheme for them

- should the rules be active straight away after inserting them via web UI or does some service/network need to be restarted?

they are active as soon as you create them

- are there any tools/commands to... say update dynamic IPs via a cron from the command lines?

you would need to manually edit IPSets in the configuration files

 

[luison]

Thanks.
So I understand then that pve-firewalls will create iptables rules. On the host affecting the KVM/CT or inside those?
Any documentation on what CHAINs pve-firewall creates?

you would need to manually edit IPSets in the configuration files

Any commands to add or remove ips from those sets from the command line (ie via cron)?

 

[shanreich]

So I understand then that pve-firewalls will create iptables rules. On the host affecting the KVM/CT or inside those?

On the host - PVE generally does not tamper with anything inside the VM. It restricts the traffic by creating firewall rules on the host for the tap interfaces.

Any documentation on what CHAINs pve-firewall creates?

It creates CHAINS with the prefix PVEFW-

edit: This might be the better place to check [5]. PVE reserves the prefixes PVEFW- / tap- / veth- / fwbr- / GROUP-

You can check by running iptables-save. The source code of the firewall module is also helpful [3].
Generally speaking, as long as you are not naming your chains with any of the prefixes mentioned above, you should be fine. You should not tamper with the chains pve-firewall creates anyway, so you don't need the exact names of the chain.

Any commands to add or remove ips from those sets from the command line (ie via cron)?

There are API endpoints [1] [2] that can be used for adding / deleting members of an IPSet.



The nftables firewall might be interesting for you, if running it is acceptable (it is currently in tech preview, so we cannot give any guarantees). Creating custom rules is easier there, because you can just create a new table. There is some info on this in the PVE documentation [4]


[1] https://pve.proxmox.com/pve-docs/api-viewer/#/cluster/firewall/ipset/{name}
[2] https://pve.proxmox.com/pve-docs/api-viewer/#/cluster/firewall/ipset/{name}/{cidr}
[3] https://git.proxmox.com/?p=pve-fire...44ba71c090ed0d8f609b8115d7d116daf252e;hb=HEAD
[4] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_usage
[5] https://git.proxmox.com/?p=pve-fire...c090ed0d8f609b8115d7d116daf252e;hb=HEAD#l1943

 

[luison]

The main need to incorporate the pve-firewall is try to limit accesses to a Docker server under a KVM.
I'll have to some testing, but I want to understand that pve-firewall will always apply root exclusively at host level by means of blocking the interface itself. Just wondering how it does to determine the interface of a specific VM of all the bridges of the host, as I understand the MAC address defined in the KVM does not necessarily correspond to that defined on the bridge at the host level.

What we are trying to do, is a mean of making sure that docker can not by any means expose a port we have not authorised previously, and make that compatible with our use of IPTABLES via CSF as we do today.

Unfortunately, docker rules are applied at prerouting level and all our current white and blacklisting are ignored!

 

[shanreich]

I'll have to some testing, but I want to understand that pve-firewall will always apply root exclusively at host level by means of blocking the interface itself.

I'm not sure I understand that sentence 100% - could you please elaborate further.

Just wondering how it does to determine the interface of a specific VM of all the bridges of the host, as I understand the MAC address defined in the KVM does not necessarily correspond to that defined on the bridge at the host level.

It doesn't go by mac address. PVE creates a tap interface [1] on the host and attaches it to the bridge / VM. So PVE knows which tap interface belongs to which VM and applies the firewall rules to all tap interfaces belonging to a VM.

You can enable MAC filtering on the VM firewall to ensure that all packets leaving that tap interface have the same MAC address as configured in the VM configuration. You can also disable mac-learning on the bridge in order to avoid the bridge learning additional MAC addresses for tap interfaces.

What we are trying to do, is a mean of making sure that docker can not by any means expose a port we have not authorised previously, and make that compatible with our use of IPTABLES via CSF as we do today.

That should be achievable by using the VM firewall with input policy DROP/REJECT and then only allowing the IP / Port combinations.

Unfortunately, docker rules are applied at prerouting level and all our current white and blacklisting are ignored!

You mean inside the VM? That doesn't matter at all then, since pve-firewall rules for VMs apply on the host, so they get evaluated before the VM sees any packet.


[1] https://en.wikipedia.org/wiki/TUN/TAP

 

[luison]

Thanks. First phrase was just wrongly expressed, sorry, I meant that "rules" are applied at host level which as you explained they are. I also understand now that firewall needs to be activated at data-center+node+vm (+ interface level too) for rules to be created.

My concern in our case is the possible conflict/redundancy between the CSF iptables rules (our current firewall) and PVE ones at the host level, but it made me understand how to centralize all firewall rules at host/interface level, which makes a lot of sense. The problem with having both is that I understand the order of creating the main chain rules would change the behaviour and I would need to have ALLOW default policy for the node and find a way to make sure that CSF are "after", so too complicated.

The reason we would like to stick with CSF is that it additionally includes additional services such as dyndns, temp banning for IPs with repeated web access tries, public blacklists, cluster with CT firewalls, etc

It doesn't go by mac address. PVE creates a tap interface [1] on the host and attaches it to the bridge / VM. So PVE knows which tap interface belongs to which VM and applies the firewall rules to all tap interfaces belonging to a VM.

Any method/command to gather that information? I mean a way to figure out from the host which is the tap interfaces that correspond to XX CT or VM? This would perhaps allow us to add manual IPTABLES and follow the same strategy.

You can enable MAC filtering on the VM firewall to ensure that all packets leaving that tap interface have the same MAC address as configured in the VM configuration. You can also disable mac-learning on the bridge in order to avoid the bridge learning additional MAC addresses for tap interfaces.

After reviewing pve-firewall generate rules, I am guessing that is already auto-created.



At the end of the day, docker's "sad" iptables nat creation rules are the ones that forces us to block potential errors somehow "in front" on a previous firewall that is not on the container/virtual machine level. This naturally should be the host but we would need a method of identifying how to block all traffic going to the VMs interface in the form that pve-firewall does.

Thanks.

 

[luison]

Still trying this for our specific environment, this is... using CSF firewall at host level and adding rules to follow pve-firewall of blocking traffic to the tap interface of the KVM or container.

It doesn't go by mac address. PVE creates a tap interface [1] on the host and attaches it to the bridge / VM. So PVE knows which tap interface belongs to which VM and applies the firewall rules to all tap interfaces belonging to a VM.

Still trying to figure out a command/s to identify the corresponding interface to a specific vm or container.

Also in doubt whether rules should be applied at FORWARD or INPUT level of that interface. We just can seem to be able to block traffic to the destination interface with a similar strategy as pve-firewall does:

#iptables -I INPUT 1 -s 77.x.x.x -m physdev --physdev-in fwln+ --physdev-is-bridged -j DROP

Either on forward... this rules just seem not applicable even if we place them as the hosts first rule.

Actually even:

# iptables -I FORWARD 1 -m physdev --physdev-in tap941i0 -j DROP
# iptables -I INPUT 1 -m physdev --physdev-in tap941i0 -j DROP
# iptables -I FORWARD 1 -i tap941i0
# iptables -I INPUT 1 -i tap941i0

Has no effect whatsoever. We've verified this is the correct interface via tcpdump -i tap941i0.
So I guess the reasons why pve-firewall can intercept traffic there but I can't must be other beyond our knowledge!

 

[shanreich]

So I guess the reasons why pve-firewall can intercept traffic there but I can't must be other beyond our knowledge!

I assume you did this with the firewall disabled? Did you make sure to use

net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1

FORWARD hook should suffice.

Also, have you checked out the nftables firewall? It can work alongside iptables rules (order of execution depending on the hook priority set in the nftables chains). It's a bit of a crutch but it might just be what could work best in your use case, since you can isolate CSF + Proxmox firewall. You can control the order of execution by utilizing the hook priority [1] feature. It's currently not possible for the nftables firewall to set the hook priority manually though without building your own package.

iptables filter rules should have priority 0, so you can use larger / smaller priorities to control whether nftables rules should be executed before / after (make sure to double-check this please, I'm not 100% sure about the intricacies).

NAT is also possible simultaneously since kernel 4.18, but I have no idea how they interact if I'm being honest. As long as you're not natting with nftables you should be fine.

[1] https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks#Priority_within_hook

 

[luison]

Thanks for feedback @shanreich
Yes pve-firewall was obviously disabled, and I was trying to add rules to our CSF generated rules to block traffic to that VM interface.

I checked:
# cat /proc/sys/net/bridge/bridge-nf-call-iptables
And it shows disabled (0) so I'm wondering now if that had to do with tests failing, which I think was the case.

So after:
# sysctl -w net.bridge.bridge-nf-call-iptables=1

I seem to be getting the result (blocked traffic) now when applying:
iptables -I FORWARD 1 -s 77.xxx.xx.201 -m physdev --physdev-in fwln+ --physdev-is-bridged -j DROP

but this might block all VMs, so I guess the correct one could just be controlling the physdev-out flag like this without needing the "fwln+"
iptables -D FORWARD -s 77.243.87.201 -m physdev --physdev-out tap941i0 --physdev-is-bridged -j DROP

I would still have to figure out how to do a proper chain, etc, but that was the base proof of concept.

I am still stuck on a way (that we could script) to determine which is the "tap" interface of X virtual machine, either by its ID, MAC, comment or other. In this case, we just guessed doing an iffconfig research.

Also wondering if this procedure would also work the same with LXC containers and how to identify their corresponding "--physdev-out" interface.

 

[luison]

So, as I've personally had been struggling with this for a very long time, this is an early implementation for a solution to limit and control open ports on a PVE host running CSF as a firewall. This is particularly helpful and intended to allow docker on the VM to manage IPTABLES without the risk of setting by mistake an open port, which by the way I still find really sad from Docker.

We created a csfpost.sh script that among other things (we do a similar strategy with NAT) creates:

# iptables -L FORWARD
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
tap941i0-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-out tap941i0 --physdev-is-bridged

All forward traffic to that interface gets routed to a chain

# iptables -L tap941i0-IN
Chain tap941i0-IN (1 references)
target     prot opt source               destination
LOCALINPUT  all  --  anywhere             anywhere
ACCEPT     all  --  192.xxx.xxx.74        anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
DROP       all  --  anywhere             anywhere

Any IP blocked or allowed in CSF LOCALINPUT chain (whitelist, dynamic ips and blocked ones) will be first checked, others will only go through ports 80 and 443. Any other public open port you need on the destination VM would have to be allowed here whatever the VMs iptables says.

We will likely adjust the order of rules and as mentioned on previous reply still need to figure how to determine which is the "tap" interface so that could be scripted too.

Now concerned about if just having
net.bridge.bridge-nf-call-iptables=1
in /etc/sysctld.d is sufficient to make it persistent. Read elsewhere some issues due to the fact that this interfaces are not loaded ad boot. Would I need to make sure this flag is configured by some other means?

 

[shanreich]

We will likely adjust the order of rules and as mentioned on previous reply still need to figure how to determine which is the "tap" interface so that could be scripted too.

Generally tap interfaces are of the form tap<vmid>i<index> - so interface net0 of a VM 1337 has the name tap1337i0.
For containers it is similar, but instead of tap it uses veth as prefix

It's a bit tricky dynamically, since you will basically have to monitor the interfaces on the host and dynamically create / delete the chains. If you really want to just jump into the CSF chain, you could just do a wildcard match on the respective firewall links like the PVE firewall does:

PVEFW-FWBR-IN  all  --  anywhere             anywhere             PHYSDEV match --physdev-in fwln+ --physdev-is-bridged

Create a rule like this, but instead of jumping into PVEFW-FWBR-IN you could just jump into LOCALINPUT and you're done. This would work for containers as well as VMs and doesn't require any dynamic creation of rules. You could then handle the other, custom, firewall rules via the Proxmox firewall and still utilize the CSF LOCALINPUT chain.

Now concerned about if just having
net.bridge.bridge-nf-call-iptables=1
in /etc/sysctld.d is sufficient to make it persistent. Read elsewhere some issues due to the fact that this interfaces are not loaded ad boot.

This should be fine, since it activates a feature that enables iptables filtering for bridged packets. It's not interface specific. Maybe check this on a test system to make 100% sure

 

[luison]

Thanks again.

So tap<ID> or veth<ID> should make it enough to just have an array of VMs and CT to apply to. Also, I can see now we could use "tap<id>+" on those rules too as a way to apply to all.

But just to understand completely, if we only wanted to limit "external" traffic would we filter --physdev-in "fwln+" or would that be vmbr0 in that case?

 

[luison]

I found a "glitch" on my setup as per my last comment.
For my solution of sending all bridged traffic to a specific interface of a VM to my own chain where I can allow only certain ports and IPs to work I realized I am forced to add a redundant line on that chain to accept outgoing traffic from that same interface, which seems like wrong as ideally only "external" traffic should enter the chain.

So for it to work I have:

# iptables-save |grep tap94
:tap941i0-IN - [0:0]
-A FORWARD -m physdev --physdev-is-in --physdev-out tap941i0 --physdev-is-bridged -j tap941i0-IN
-A tap941i0-IN -p tcp -m tcp --dport 443 -m comment --comment "Puerto permitido globalmente" -j ACCEPT
-A tap941i0-IN -p tcp -m tcp --dport 80 -m comment --comment "Puerto permitido globalmente" -j ACCEPT
-A tap941i0-IN -s 192.xxx.39.117/32 -j ACCEPT
-A tap941i0-IN -m physdev --physdev-out tap941+ -j ACCEPT
-A tap941i0-IN
-A tap941i0-IN -m comment --comment "Evaluación de LOCALINPUT" -j LOCALINPUT
-A tap941i0-IN -m comment --comment "Bloqueo de tráfico restante" -j DROP

When I tend to think there must be a way to limit

-A FORWARD -m physdev --physdev-is-in --physdev-out tap941i0 --physdev-is-bridged -j tap941i0-IN

to only affect incoming traffic and not outgoing from that interface, as all bridged traffic is allowed by default. But I can't seem to figure this out... tried to add that the "-physdev-in fwln+" as how pve-firewall seems to define it, but I get no result. Also tried using vmbr0.

So unless I add to the chain:
-A tap941i0-IN -m physdev --physdev-out tap941+ -j ACCEPT
to the chain, outgoing traffic seems to get blocked.

Unclear what to add to my FORWARD rule so it only affects incoming (bridged traffic from external connections) and not originating on that interface.

 

[shanreich]

Do you have firewall=1 set in your network interface configuration?

 

[luison]

Do you have firewall=1 set in your network interface configuration?

It was yes, and that seemed to be, until further research, the issue. What I still don't get is that I assumed that if pve-firewall was disabled at datacenter/host level all this configurations on the containers and vms did not have any effect, which I was obviously wrong.

I also don't completely get why that firewall=1 was affecting outgoing traffic, but after disabling it the basic (no input dev defined)
-A FORWARD -m physdev --physdev-is-in --physdev-out tap941i0 --physdev-is-bridged -j tap941+

rule to push all forwarded traffic to my chain, seems to work and the vm has outgoing traffic again.
thanks

 

[shanreich]

This setting is a bit misleading tbh, since what it currently does is create a firewall bridge when the interface is attached to the VM. The firewall bridge is necessary for the old firewall to function.

 

[luison]

until further research

Luckily, I said that! Unfortunately, I was not write and not sure why we thought the issue was solved after deactivating the firewall option on the interface. As we say in Spanish, claimed my victory too soon

It did not make sense in any case, as I get now that the option creates the fw*** interfaces but not necessarily ads any rules if the pve-firewall is disabled.

So back to the source problem and trying to analyse the behaviour of pve-firewall on these to try to understand/solve our case. Basically, our "incoming tap interface conditions" also affect (can't figure why) the outgoing traffic on the VM. My understanding for traffic conditions for bridged are limited, so I will need to get pve.firewall again on a vm o test server to figure out.

I did notice on the saved one that pve creates an "OUT" chain per interface too but it marks traffic to be accepted or not at a later rule. I still think there should be a way to apply our chain only to "incoming" traffic to that interface but not the one originating on it.

 

[shanreich]

-A FORWARD -m physdev --physdev-is-in --physdev-out tap941i0 --physdev-is-bridged -j tap941+

Are you sure that this is the proper rule? What about something like this (removing the physdev-is-in)?

-A FORWARD -m physdev --physdev-out tap941i0 --physdev-is-bridged -j tap941+

Is the plus intended? Because that is usually a wildcard, but you are using it in the jump parameter, where no wildcards are applicable

 

[luison]

This is currently our iptables related logic which coincides I understand with your suggestion and only applies to our first interface which is the public one. We tried various alternatives for in/out dev with no different result:

# iptables-save |grep  tap941i0-IN
:tap941i0-IN - [0:0]
-A FORWARD -m physdev --physdev-out tap941i0 --physdev-is-bridged -j tap941i0-IN
-A tap941i0-IN -p tcp -m tcp --dport 443 -m comment --comment "Puerto permitido globalmente" -j ACCEPT
-A tap941i0-IN -p tcp -m tcp --dport 80 -m comment --comment "Puerto permitido globalmente" -j ACCEPT
-A tap941i0-IN -s 79.xxx.xx.xx/32 -m comment --comment "IP permitida para pruebas" -j ACCEPT
-A tap941i0-IN -m comment --comment "Evaluación de LOCALINPUT" -j LOCALINPUT
-A tap941i0-IN -m comment --comment "Bloqueo de tráfico restante" -j DROP

Works great blocking connections, but we seem to lose connectivity from the VM console, so can't even ping external IP. We can with local ones as I understand they route through tap941i1. As I see it... the outgoing traffic is also passing through that same chain, so we either filter it on the condition in FORWARD or add an ACCEPT that applies only to outgoing connections.

Currently, researching this from the pve-firewall saved rules when it was active:
[479:33641] -A PVEFW-FWBR-OUT -m physdev --physdev-in tap941i0 --physdev-is-bridged -j tap941i0-OUT

As mentioned, I can't seem to understand the why of "--physdev-in" in this case.
The complete chain is (was when active) this:

:tap941i0-OUT - [0:0]
[479:33641] -A PVEFW-FWBR-OUT -m physdev --physdev-in tap941i0 --physdev-is-bridged -j tap941i0-OUT
[0:0] -A tap941i0-OUT -m mac ! --mac-source 02:00:00:dd:11:80 -j DROP
[479:33641] -A tap941i0-OUT -j MARK --set-xmark 0x0/0x80000000
[479:33641] -A tap941i0-OUT -j GROUP-allow_nosotros-OUT
[0:0] -A tap941i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
[479:33641] -A tap941i0-OUT -j GROUP-webservers-OUT
[0:0] -A tap941i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
[479:33641] -A tap941i0-OUT -g PVEFW-SET-ACCEPT-MARK
[0:0] -A tap941i0-OUT -m comment --comment "PVESIG:O2eMuG4u7M592CngIIBkqL+yLt4"