Proxmox VE 8.3 Masquerading (NAT) with nftables (not iptables)

K

khoneini

New Member
Dec 18, 2024
4
0
1
Decided yesterday to update my Proxmox VE node to version 8.3.2 and realised quickly that the way forwarding traffic works has changed as Proxmox seems to have made its way to nftables as of Proxmox VE 8.3.

As outlined in the Firewall documentation, forwarded traffic is currently only possible when using the new nftables-based proxmox-firewall and any forward rules previously setup and used by pve-firewall (Proxmox's iptables-based firewall) will have no effect.

This lead to the following problem which is that for the VM's that only have a private IP (LAN), are as of this update, not able to route traffic over the Proxmox host interface which I'm using as WAN. I had used the Masquerading (NAT) with iptables configuration which no longer works with proxmox-firewall.

Long story short, I'm not familiar with nftables and need to apply the same forwarding rules with masquerading in proxmox-firewall. Specifically this part:
Code: 
post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

The postrouting should be fairly easy in nftables:
Code: 
# Define the NAT table
table ip nat {
    # Define the postrouting chain
    chain postrouting {
        type nat hook postrouting priority 100; policy accept;
        # Masquerade rule
        oifname "eno1" masquerade
    }
}

but I'm struggling getting the prerouting to work with conntrack zones which are needed for the outgoing connections. Any help is appreciated as I've been trying to get this to work for the past couple of days now.
 
Last edited:
Additionally I've tried to use iptables-save. Which generated the following output. To avoid confusion, eno1 is in my config enp7s0.4000:
iptables.txt
Code: 
# Generated by iptables-save v1.8.9 on Wed Dec 18 13:02:39 2024
*filter
:INPUT ACCEPT [329678:40633531]
:FORWARD ACCEPT [1710251:303955058]
:OUTPUT ACCEPT [438881:66041429]
COMMIT
# Completed on Wed Dec 18 13:02:39 2024
# Generated by iptables-save v1.8.9 on Wed Dec 18 13:02:39 2024
*raw
:PREROUTING ACCEPT [2042685:343515198]
:OUTPUT ACCEPT [443482:66466773]
-A PREROUTING -i fwbr+ -j CT --zone 1
COMMIT
# Completed on Wed Dec 18 13:02:39 2024
# Generated by iptables-save v1.8.9 on Wed Dec 18 13:02:39 2024
*nat
:PREROUTING ACCEPT [324471:19731833]
:INPUT ACCEPT [67521:5767972]
:OUTPUT ACCEPT [365:27077]
:POSTROUTING ACCEPT [257172:13981396]
-A POSTROUTING -s 10.10.0.0/20 -o enp7s0.4000 -j MASQUERADE
COMMIT
# Completed on Wed Dec 18 13:02:39 2024

ruleset.nft
Code: 
# Translated by iptables-restore-translate v1.8.9 on Wed Dec 18 13:03:48 2024
add table ip filter
add chain ip filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip filter FORWARD { type filter hook forward priority 0; policy accept; }
add chain ip filter OUTPUT { type filter hook output priority 0; policy accept; }
add table ip raw
add chain ip raw PREROUTING { type filter hook prerouting priority -300; policy accept; }
add chain ip raw OUTPUT { type filter hook output priority -300; policy accept; }
# -t raw -A PREROUTING -i fwbr+ -j CT --zone 1
add table ip nat
add chain ip nat PREROUTING { type nat hook prerouting priority -100; policy accept; }
add chain ip nat INPUT { type nat hook input priority 100; policy accept; }
add chain ip nat OUTPUT { type nat hook output priority -100; policy accept; }
add chain ip nat POSTROUTING { type nat hook postrouting priority 100; policy accept; }
add rule ip nat POSTROUTING oifname "enp7s0.4000" ip saddr 10.10.0.0/20 counter masquerade
# Completed on Wed Dec 18 13:03:48 2024

When trying to apply the ruleset.nft it doesn't seem to apply the PREROUTING config. The reason for that is that iptables-save converted the rule to a wrong nftables syntax. I therefore tried the following 2 nftables rules which have a correct syntax:
Code: 
nft add rule ip nat PREROUTING iifname "fwbr*" ct zone set 1
nft add rule ip raw PREROUTING iifname "fwbr*" ct zone set 1

My nftables config looks now like:
Code: 
table ip nat {
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        ip saddr 10.10.0.0/20 oif "enp7s0.4000" masquerade
    }

    chain PREROUTING {
        type nat hook prerouting priority dstnat; policy accept;
        iifname "fwbr*" ct zone set 1
    }

    chain INPUT {
        type nat hook input priority 100; policy accept;
    }

    chain OUTPUT {
        type nat hook output priority -100; policy accept;
    }

    chain POSTROUTING {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "enp7s0.4000" ip saddr 10.10.0.0/20 counter packets 0 bytes 0 masquerade
    }
}
table ip raw {
    chain prerouting {
        type filter hook prerouting priority raw; policy accept;
    }

    chain PREROUTING {
        type filter hook prerouting priority raw; policy accept;
        iifname "fwbr*" ct zone set 1
    }

    chain OUTPUT {
        type filter hook output priority raw; policy accept;
    }
}
table ip filter {
    chain INPUT {
        type filter hook input priority filter; policy accept;
    }

    chain FORWARD {
        type filter hook forward priority filter; policy accept;
    }

    chain OUTPUT {
        type filter hook output priority filter; policy accept;
    }
}
However my VM, attached to the bridge adapter (vmbr1), still can't reach the internet.

Additionally I do have nftables (tech preview) enabled in: Datacenter > Node > Firewall > Options
As well as having set the Forward Policy to ACCEPT in: Datacenter > Firewall > Options

Any help is appreciated, thanks a lot!
 
Realised my nftables was overwritten by applying the ruleset.nft. I flushed the nftables and it went back to the default config of /etc/nftables.conf which is the proxmox-firewall default configuration. I'm curious to which chains I should apply the PREROUTING and POSTROUTING rules from my iptables:
Code: 
table inet filter {
    chain input {
        type filter hook input priority filter; policy accept;
    }

    chain forward {
        type filter hook forward priority filter; policy accept;
    }

    chain output {
        type filter hook output priority filter; policy accept;
    }
}
table inet proxmox-firewall {
    set v4-dc/management {
        type ipv4_addr
        flags interval
        auto-merge
    }

    set v4-dc/management-nomatch {
        type ipv4_addr
        flags interval
        auto-merge
    }

    set v6-dc/management {
        type ipv6_addr
        flags interval
        auto-merge
    }

    set v6-dc/management-nomatch {
        type ipv6_addr
        flags interval
        auto-merge
    }

    set v4-synflood-limit {
        type ipv4_addr
        flags dynamic,timeout
        timeout 1m
    }

    set v6-synflood-limit {
        type ipv6_addr
        flags dynamic,timeout
        timeout 1m
    }

    map bridge-map {
        type ifname : verdict
    }

    chain do-reject {
        meta pkttype broadcast drop
        ip saddr 224.0.0.0/4 drop
        meta l4proto tcp reject with tcp reset
        meta l4proto { icmp, ipv6-icmp } reject
        reject with icmp host-prohibited
        reject with icmpv6 admin-prohibited
        drop
    }

    chain accept-management {
        ip saddr @v4-dc/management ip saddr != @v4-dc/management-nomatch accept
        ip6 saddr @v6-dc/management ip6 saddr != @v6-dc/management-nomatch accept
    }

    chain block-synflood {
        tcp flags != syn / fin,syn,rst,ack return
        jump ratelimit-synflood
        drop
    }

    chain log-drop-invalid-tcp {
        jump log-invalid-tcp
        drop
    }

    chain block-invalid-tcp {
        tcp flags fin,psh,urg / fin,syn,rst,psh,ack,urg goto log-drop-invalid-tcp
        tcp flags ! fin,syn,rst,psh,ack,urg goto log-drop-invalid-tcp
        tcp flags syn,rst / syn,rst goto log-drop-invalid-tcp
        tcp flags fin,syn / fin,syn goto log-drop-invalid-tcp
        tcp sport 0 tcp flags syn / fin,syn,rst,ack goto log-drop-invalid-tcp
    }

    chain allow-ndp-in {
        icmpv6 type { nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect } accept
    }

    chain block-ndp-in {
        icmpv6 type { nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect } drop
    }

    chain allow-ndp-out {
        icmpv6 type { nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert } accept
    }

    chain block-ndp-out {
        icmpv6 type { nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert } drop
    }

    chain block-conntrack-invalid {
        ct state invalid drop
    }

    chain block-smurfs {
        ip saddr 0.0.0.0 return
        meta pkttype broadcast goto log-drop-smurfs
        ip saddr 224.0.0.0/4 goto log-drop-smurfs
    }

    chain allow-icmp {
        icmp type { destination-unreachable, source-quench, time-exceeded } accept
        icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem } accept
    }

    chain log-drop-smurfs {
        jump log-smurfs
        drop
    }

    chain default-in {
        iifname "lo" accept
        jump allow-icmp
        ct state established,related accept
        meta l4proto igmp accept
        tcp dport { 22, 3128, 5900-5999, 8006 } jump accept-management
        udp dport 5405-5412 accept
        udp dport { 135, 137-139, 445 } goto do-reject
        udp sport 137 udp dport 1024-65535 goto do-reject
        tcp dport { 135, 139, 445 } goto do-reject
        udp dport 1900 drop
        udp sport 53 drop
    }

    chain default-out {
        oifname "lo" accept
        jump allow-icmp
        ct state vmap { invalid : drop, established : accept, related : accept }
    }

    chain before-bridge {
        meta protocol arp accept
        meta protocol != arp ct state vmap { invalid : drop, established : accept, related : accept }
    }

    chain host-bridge-input {
        type filter hook input priority filter - 1; policy accept;
        iifname vmap @bridge-map
    }

    chain host-bridge-output {
        type filter hook output priority filter + 1; policy accept;
        oifname vmap @bridge-map
    }

    chain input {
        type filter hook input priority filter; policy accept;
        jump default-in
        jump ct-in
        jump option-in
        jump host-in
        jump cluster-in
    }

    chain output {
        type filter hook output priority filter; policy accept;
        jump default-out
        jump option-out
        jump host-out
        jump cluster-out
    }

    chain forward {
        type filter hook forward priority filter; policy accept;
        jump host-forward
        jump cluster-forward
    }

    chain ratelimit-synflood {
    }

    chain log-invalid-tcp {
    }

    chain log-smurfs {
    }

    chain option-in {
    }

    chain option-out {
    }

    chain cluster-in {
    }

    chain cluster-out {
    }

    chain host-in {
    }

    chain host-out {
    }

    chain cluster-forward {
    }

    chain host-forward {
    }

    chain ct-in {
    }
}
table bridge proxmox-firewall-guests {
    map vm-map-in {
        typeof oifname : verdict
    }

    map vm-map-out {
        typeof iifname : verdict
    }

    map bridge-map {
        type ifname . ifname : verdict
    }

    chain allow-dhcp-in {
        udp sport . udp dport { 547 . 546, 67 . 68 } accept
    }

    chain allow-dhcp-out {
        udp sport . udp dport { 546 . 547, 68 . 67 } accept
    }

    chain block-dhcp-in {
        udp sport . udp dport { 547 . 546, 67 . 68 } drop
    }

    chain block-dhcp-out {
        udp sport . udp dport { 546 . 547, 68 . 67 } drop
    }

    chain allow-ndp-in {
        icmpv6 type { nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect } accept
    }

    chain block-ndp-in {
        icmpv6 type { nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, nd-redirect } drop
    }

    chain allow-ndp-out {
        icmpv6 type { nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert } accept
    }

    chain block-ndp-out {
        icmpv6 type { nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert } drop
    }

    chain allow-ra-out {
        icmpv6 type { nd-router-advert, nd-redirect } accept
    }

    chain block-ra-out {
        icmpv6 type { nd-router-advert, nd-redirect } drop
    }

    chain allow-icmp {
        icmp type { destination-unreachable, source-quench, time-exceeded } accept
        icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem } accept
    }

    chain do-reject {
        meta pkttype broadcast drop
        ip saddr 224.0.0.0/4 drop
        meta l4proto tcp reject with tcp reset
        meta l4proto { icmp, ipv6-icmp } reject
        reject with icmp host-prohibited
        reject with icmpv6 admin-prohibited
        drop
    }

    chain pre-vm-out {
        meta protocol != arp ct state vmap { invalid : drop, established : accept, related : accept }
    }

    chain vm-out {
        type filter hook prerouting priority 0; policy accept;
        jump allow-icmp
        iifname vmap @vm-map-out
    }

    chain pre-vm-in {
        meta protocol != arp ct state vmap { invalid : jump invalid-conntrack, established : accept, related : accept }
        meta protocol arp accept
    }

    chain vm-in {
        type filter hook postrouting priority 0; policy accept;
        jump allow-icmp
        oifname vmap @vm-map-in
    }

    chain before-bridge {
        meta protocol arp accept
        meta protocol != arp ct state vmap { invalid : drop, established : accept, related : accept }
    }

    chain forward {
        type filter hook forward priority 0; policy accept;
        meta ibrname . meta obrname vmap @bridge-map
    }

    chain invalid-conntrack {
    }
}
 
Pretty lost still.. if anyone has any clues feel free to drop them as I'm running out of things to try..
 
I know it's been a bit since you first posted this but I had to do the same thing you do and here are the configuration lines I added to /etc/nftables.conf to do so.

Code: 
table inet nat {
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
    }

    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "vmbr2" ip daddr != 192.168.2.0/24 counter packets 238 bytes 14876 snat ip to 10.0.200.57
    }
}

Note that vmbr2 is my external VLAN bridge that I want the NAT traffic to use, 192.168.2.0/24 is the local only subnet, ie within the hypervisor, and 10.0.200.57 is the IP address on the vmbr2 bridge. I was able to do this by using the iptables-translate command and a how-to about iptables with SNAT forwarding.
One issue I do have with this is that nftables does not load the rules from /etc/nftables.conf on boot. I'm still tracking down that issue but running "nft -f /etc/nftables" will reload the rules.

Edit: formating
 
Last edited:
You must log in or register to reply here.
Top Bottom