[IPSET ipfilter-net0] - not working unless I add vm specific firewall rule to only allow traffic from the IPSET

D

David123

New Member
Proxmox Subscriber
Jul 6, 2024
18
2
3
Hello

I have an issue with the way https://pve.proxmox.com/pve-docs/chapter-pve-firewall.html#pve_firewall_ipfilter_section is working. First I'd like to clarify that Firewall is enabled at DC, Server and VM level.
VM firewall options:
firewall setting.png

Focusing on the /etc/pve/firewall# cat 584.fw output/settings:
If I have this firewall config, then vm will still be able to set any IP at their network interface such as 95.12.124.156 and get traffic from it:
Code: 
/etc/pve/firewall# cat 584.fw
[OPTIONS]

enable: 1
policy_out: ACCEPT
ipfilter: 1
policy_in: ACCEPT

[IPSET ipfilter-net0]

95.12.124.154 # Assigned
::1 # Interface with no v6 IPs

I managed to get IP filtering to work by setting "policy_in: DROP" and then adding a firewall rule that would only allow traffic from the IPSET ipfilter-net0. But surely this should not be required?

Code: 
/etc/pve/firewall# cat 584.fw
[OPTIONS]
enable: 1
policy_out: ACCEPT
ipfilter: 1
policy_in: DROP

[IPSET ipfilter-net0]

95.12.124.154 # Assigned
::1 # Interface with no v6 IPs

[RULES]

IN ACCEPT -dest +guest/ipfilter-net0 -log nolog


VM is set to use vmbr0 as bridge and I have this network config at my proxmox install:
Code: 
/etc/network# cat interfaces
auto lo
iface lo inet loopback

iface enp67s0f1np1 inet manual
    mtu 9000

iface enp67s0f0np0 inet manual
    mtu 9000

auto bond0
iface bond0 inet manual
    bond-slaves enp67s0f0np0 enp67s0f1np1
    bond-mode 802.3ad
    bond-miimon 100
    bond-downdelay 200
    bond-updelay 200
    bond-lacp-rate 1
    mtu 9000

auto vmbr0
iface vmbr0 inet manual
    mtu 9000
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    address 95.12.124.9/24
    gateway 95.12.124.1
    vlan-raw-device bond0
    bridge_vlan_aware yes
    bridge-vids 1000 500

iface vmbr0 inet6 manual
    mtu 9000
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    address 2001:241::9/36
    gateway 2001:241::1
    vlan-raw-device bond0
    bridge_vlan_aware yes
    bridge-vids 1000 500

auto vmbr0.500
iface vmbr0.500 inet static
    mtu 9000
    address 10.0.0.9/24
    gateway 10.0.0.1
    vlan-raw-device bond0

auto vmbr0.1000
iface vmbr0.1000 inet manual
    vlan-raw-device vmbr0

source /etc/network/interfaces.d/*

I need help fixing / understanding why ipfilter is not working on my proxmox server.

Many thanks!
 
Last edited:
IP Filtering only applies to outgoing traffic:

These filters belong to a VM’s network interface and are mainly used to prevent IP spoofing. If such a set exists for an interface then any outgoing traffic with a source IP not matching its interface’s corresponding ipfilter set will be dropped.
Click to expand...
 
  • Like
Reactions: David123
Hey, It seems like ipfilter is working, but it requires this additional step with the rule to block traffic outside assigned IPs, The default firewall setting policy_in: ACCEPT allows everything, so after switching to DROP, you need to add a rule allowing the correct traffic.Also, I recommend double-checking if all addresses are properly assigned in ipset, Let me know if you need further help with the config! :)
 
Seems like this is a bug with '-nomatch' logic.
By enabling ipfilter we get nft rule like this:

Code: 
    iifname "tap102i0" ip saddr != @v4-guest-102/ipfilter-net0 ip saddr @v4-guest-102/ipfilter-net0-nomatch drop

or in other words:

Code: 
  IF (iifname "tap102i0") AND (ip saddr != @v4-guest-102/ipfilter-net0) AND (ip saddr @v4-guest-102/ipfilter-net0-nomatch) THEN drop

But the last condition will NEVER be true because ipfilter-net0-nomatch is ALWAYS empty.
 
But.... Seems like i found a workaround.

We need to insert in ipfilter_net0 something like this:

Code: 
1.2.3.4 <- legitimate address
! 0.0.0.0/1 <- NOT match
! 128.0.0.0/1 <- NOT match
 
Remidor said:
Seems like this is a bug with '-nomatch' logic.
By enabling ipfilter we get nft rule like this:

Code: 
    iifname "tap102i0" ip saddr != @v4-guest-102/ipfilter-net0 ip saddr @v4-guest-102/ipfilter-net0-nomatch drop

or in other words:

Code: 
  IF (iifname "tap102i0") AND (ip saddr != @v4-guest-102/ipfilter-net0) AND (ip saddr @v4-guest-102/ipfilter-net0-nomatch) THEN drop

But the last condition will NEVER be true because ipfilter-net0-nomatch is ALWAYS empty.
Click to expand...

Thanks for the report! I'll look into fixing this soon. Could you open a bug in Bugzilla (https://bugzilla.proxmox.com) ?
 
shanreich said:
Thanks for the report! I'll look into fixing this soon. Could you open a bug in Bugzilla (https://bugzilla.proxmox.com) ?
Click to expand...

Hi, Shanreich!

I have some not so good news. There is a more serious bug.

According to https://github.com/proxmox/proxmox-firewall/blob/master/proxmox-firewall/src/firewall.rs, proxmox-firewall, when reloading rules, it does the following:
1. Flushes all rules.
2. Constructs and loads new ones.
Unfortunately, nothing good will come of this.
If we do this, we have a situation in which, for a very short but noticeable time, the rules are completely absent.

While in the virtual machine, we can do something like this:

Code: 
python3 -c 'from scapy.all import send, IP, ICMP; send(IP(src="10.172.222.10", dst="10.172.222.11")/ICMP(id=2222, seq=42), count=1000000)'

EDIT: (Sorry. 10.172.222.10 and 10.172.222.11 are external addresses for the virtual machine. This is packet spoofing. But as will become clear below, this is not the only scenario.)

And in that millisecond when there are no rules, the packet will pass. This means that it will be listed in conntrack as ESTABLISHED. And this means that even when the rules come back, packets will still pass even if they should be dropped.

I have no simple workaround for this. :(
On my setup, I can bypass ipfilter (for example!) after several thousand (or tens of thousands) of packets.

In the case of ipfilter, it would help to swap the rules:

Code: 
        chain guest-138-out {
                iifname "tap138i0" ip saddr != @v4-guest-138/ipfilter-net0 ip saddr @v4-guest-138/ipfilter-net0-nomatch drop
                iifname "tap138i0" ip6 saddr != @v6-guest-138/ipfilter-net0 ip6 saddr @v6-guest-138/ipfilter-net0-nomatch drop
                iifname "tap138i0" arp saddr ip != @v4-guest-138/ipfilter-net0 drop
                iifname . ether saddr != { "tap138i0" . bc:24:11:67:93:96 } drop
                iifname . arp saddr ether != { "tap138i0" . bc:24:11:67:93:96 } drop
                jump pre-vm-out

instead of:

Code: 
        chain guest-138-out {
                jump pre-vm-out
                iifname "tap138i0" ip saddr != @v4-guest-138/ipfilter-net0 ip saddr @v4-guest-138/ipfilter-net0-nomatch drop
                iifname "tap138i0" ip6 saddr != @v6-guest-138/ipfilter-net0 ip6 saddr @v6-guest-138/ipfilter-net0-nomatch drop
                iifname "tap138i0" arp saddr ip != @v4-guest-138/ipfilter-net0 drop
                iifname . ether saddr != { "tap138i0" . bc:24:11:67:93:96 } drop
                iifname . arp saddr ether != { "tap138i0" . bc:24:11:67:93:96 } drop

but in the case of other rules this will not help, and is impossible.
I mean, EVEN policy is getting through.

You can make sure that proxmox-firewall is to blame in a simple way:

Code: 
nft list ruleset > nft.rules
systemctl stop proxmox-firewall pve-firewall
nft -f nft.rules

Once this is done, there is no way to bypass the filter, even with the same order of rules.

------
Best regards,
Remidor
 
Last edited:
Remidor said:
According to https://github.com/proxmox/proxmox-firewall/blob/master/proxmox-firewall/src/firewall.rs, proxmox-firewall, when reloading rules, it does the following:
1. Flushes all rules.
2. Constructs and loads new ones.
Unfortunately, nothing good will come of this.
If we do this, we have a situation in which, for a very short but noticeable time, the rules are completely absent.
Click to expand...

That might be a bug in nftables then, because those statements should be atomic - at least according to [1] - and we're relying on that behavior. So if that works then that's actually a bug in nftables itself. I'll check whether I can reproduce that on my machine as well.

[1] https://wiki.nftables.org/wiki-nftables/index.php/Atomic_rule_replacement
 
shanreich said:
That might be a bug in nftables then, because those statements should be atomic - at least according to [1] - and we're relying on that behavior. So if that works then that's actually a bug in nftables itself. I'll check whether I can reproduce that on my machine as well.

[1] https://wiki.nftables.org/wiki-nftables/index.php/Atomic_rule_replacement
Click to expand...

Sorry, but function full_host_fw() (https://github.com/proxmox/proxmox-firewall/blob/master/proxmox-firewall/src/firewall.rs#L214) doesn't look like an atomic operation. :( I admit that I misunderstood something - Rust is not my strong point.
I am very grateful for your help.

--------
Best regards,
Remidor
 
Remidor said:
Sorry, but function full_host_fw() (https://github.com/proxmox/proxmox-firewall/blob/master/proxmox-firewall/src/firewall.rs#L214) doesn't look like an atomic operation. :( I admit that I misunderstood something - Rust is not my strong point.
Click to expand...

It only generates the JSON representation of the ruleset, the functions merely create structs that represent the ruleset. The whole ruleset is passed to the nft userspace tool here [1]. Please, in the future if you think you have found an issue like this use our security reporting channels [2] rather than the public forum.

[1] https://git.proxmox.com/?p=proxmox-...984b5fa2aae206ac14837210fdd950a47;hb=HEAD#l48
[2] https://pve.proxmox.com/wiki/Security_Reporting
 
Last edited:
You must log in or register to reply here.
Top Bottom