nftables: no stateful rule for output

[Gilou]

Hi,
I wanted to try nftables on Proxmox, it seems quite nicely done, bravo!

I guess most users don't use any output filters, but if using them in iptables, we get a stateful output rule, allowing to only open INPUT for a given port, and assume that it will go out.

Chain PVEFW-HOST-OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination       
 283K 1225M ACCEPT     0    --  *      lo      0.0.0.0/0            0.0.0.0/0         
    0     0 DROP       0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
5507K   38G ACCEPT     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED

In nftables, we do get a stateful rule for input (through default-in / default-in: ct state established,related accept), but it doesn't seem to be set for output!
It is available in default-out, but VMs packets don't jump to that chain it seems (EDIT#2, no they don't, because of the priority 0 + goto). So, maybe, allow for a jump to default-out before drop for VMs ? Or set a specific default for them.

It's not necessarily a bad idea to disable the stateful firewall (or having any impact on that), but as it is, it does not work the same way the pve-firewall using iptables does!

Cheers,
Gilou

 

[Gilou]

To summarize the issue, and how to reproduce it.

This test was done using IPv6 on WAN, and IPv4 on LAN.

- set a CT/VM to policy DROP/DROP
- enable port 22 INPUT

On iptables/pve-firewall, conf set to:

[OPTIONS]

enable: 1
policy_out: DROP

[RULES]

IN SSH(ACCEPT) -log nolog

ssh to_ct: SYN, SYN/ACK, DATA, all is fine.

On nftables/proxmox-firewall, conf is the same, but host has the tech preview/nftables setting to 1:
Using IPv6: With default settings, that allows NDP, I get the router's MAC, but:
SYN => OK, SYN/ACK goes out, but is filtered out.

Using IPv4, no MAC, no ARP (I see the request, CT answers it, but it doesn't go out, not reaching the destination, so… nothing happens).

Of course, if I add a rule to enable outgoing packets from port 22, it works. But that doesn't feel very stateful?

 

[encryptedserver]

I have a similar problem https://forum.proxmox.com/threads/b...s-not-work-accept-ping-using-nftables.145748/

Pretty common and basic firewall rule. Allow gateway and block LAN, IPv4 is somewhat working IPv6 not at all.

 

[Gilou]

Well, your rules end up with… accept, so it's probably not the same issue, as connection state is probably not directly your issue, I'd say

 

[encryptedserver]

It used to work with iptables but not with the new nftables. I am not sure about creating new rules in nftables.

 

[Gilou]

There are two parts I'm interested in in the nft rulesets:

        chain output {
                type filter hook output priority filter; policy accept;
                jump default-out
                jump option-out
                jump host-out
                jump cluster-out
        }
        chain default-out {
                oifname "lo" accept
                ct state vmap { invalid : drop, established : accept, related : accept }
        }

and then:

       chain vm-out {
                type filter hook prerouting priority 0; policy accept;
                iifname vmap @vm-map-out
        }

That leads to vm-map-out, that does a _goto_ to guest-xxx-out. The -out policy does neither jump to default-out, nor to anything enabling established connections.

I'm not too experienced with nftables, but since there is a priority "filter" that should translate to 0 for the default rule, and that chain is hooked in output, I'm guessing the -out chain mapping, hooked in pre-routing, has priority. And does no jump to default-out, so hits the final drop like a wall… thus… not very nice to us.

We need to add a jump to default-out in the guest-xxx-out rule, I'd say early, or set a separate default-vm-out that does the stateful dance?

 

[Gilou]

Solution, as it's not very configurable as it is, change proxmox-firewall/resources/proxmox-firewall.nft so that for vm-out it reads:

chain vm-out {
        type filter hook prerouting priority 0; policy accept;
        ct state related,established accept
        iifname vmap @vm-map-out
    }

 

[Gilou]

This is addressed in https://forum.proxmox.com/threads/n...vm-theres-no-way-to-accept-arp-output.146015/

 

[Gilou]

What may not be addressed however, is the fact that the DHCP/NDP rules seem to be set backwards, preventing "client" service when set to no in the options, when the iptables system did it "correctly". There may be room for improvement in the docs, mentionning that DHCP means in fact DHCP server, and NDP being enabling DHCPv6 and router solicitations.