SDN Firewall rules explanation needed

[koniambo]

Hi,

I'm testing SDN and firewalling and I have a problem with SDN firewall rules, I don't understand how they apply. I want to drop all traffic in my vlan 36 except for VM insinde vlan 36 but the policy forward drop override everything.

cat /etc/pve/sdn/firewall/Vlan36.fw
[OPTIONS]

policy_forward: DROP
log_level_forward: debug
enable: 1

[RULES]

FORWARD ACCEPT -source +sdn/Vlan36-all -dest +sdn/Vlan36-all -log debug

If someone has an idea on how to do SDN firewall

 

[shanreich]

Can you post the output of the following command?

nft list ruleset

Are there any errors logged in the firewall daemon?

systemctl status proxmox-firewall

 

[koniambo]

Thanks for your help

Proxmox firewall is runing well, but the vnets -> rules in the firewall is not used. It seems that only the vnets -> options -> forward policy matters
I tried to had some rules to test and nothings happens
this is the output of nft list ruleset

chain bridge-Vlan36 {
jump before-bridge
ip saddr @v4-sdn/Vlan36-all ip saddr != @v4-sdn/Vlan36-all-nomatch ip daddr @v4-sdn/Vlan36-all ip daddr != @v4-sdn/Vlan36-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip6 saddr @v6-sdn/Vlan36-all ip6 saddr != @v6-sdn/Vlan36-all-nomatch ip6 daddr @v6-sdn/Vlan36-all ip6 daddr != @v6-sdn/Vlan36-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip saddr @v4-sdn/Vlan36-all ip saddr != @v4-sdn/Vlan36-all-nomatch ip daddr @v4-sdn/Vlan36-all ip daddr != @v4-sdn/Vlan36-all-nomatch accept
ip6 saddr @v6-sdn/Vlan36-all ip6 saddr != @v6-sdn/Vlan36-all-nomatch ip6 daddr @v6-sdn/Vlan36-all ip6 daddr != @v6-sdn/Vlan36-all-nomatch accept
ip saddr @v4-sdn/Vlan36-all ip saddr != @v4-sdn/Vlan36-all-nomatch ip daddr @v4-sdn/Vlan35-all ip daddr != @v4-sdn/Vlan35-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip6 saddr @v6-sdn/Vlan36-all ip6 saddr != @v6-sdn/Vlan36-all-nomatch ip6 daddr @v6-sdn/Vlan35-all ip6 daddr != @v6-sdn/Vlan35-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip saddr @v4-sdn/Vlan36-all ip saddr != @v4-sdn/Vlan36-all-nomatch ip daddr @v4-sdn/Vlan35-all ip daddr != @v4-sdn/Vlan35-all-nomatch accept
ip6 saddr @v6-sdn/Vlan36-all ip6 saddr != @v6-sdn/Vlan36-all-nomatch ip6 daddr @v6-sdn/Vlan35-all ip6 daddr != @v6-sdn/Vlan35-all-nomatch accept
ip saddr @v4-sdn/Vlan35-all ip saddr != @v4-sdn/Vlan35-all-nomatch ip daddr @v4-sdn/Vlan35-all ip daddr != @v4-sdn/Vlan35-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip6 saddr @v6-sdn/Vlan35-all ip6 saddr != @v6-sdn/Vlan35-all-nomatch ip6 daddr @v6-sdn/Vlan35-all ip6 daddr != @v6-sdn/Vlan35-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip saddr @v4-sdn/Vlan35-all ip saddr != @v4-sdn/Vlan35-all-nomatch ip daddr @v4-sdn/Vlan35-all ip daddr != @v4-sdn/Vlan35-all-nomatch accept
ip6 saddr @v6-sdn/Vlan35-all ip6 saddr != @v6-sdn/Vlan35-all-nomatch ip6 daddr @v6-sdn/Vlan35-all ip6 daddr != @v6-sdn/Vlan35-all-nomatch accept
ip saddr @v4-sdn/Vlan35-all ip saddr != @v4-sdn/Vlan35-all-nomatch ip daddr @v4-sdn/Vlan36-all ip daddr != @v4-sdn/Vlan36-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip6 saddr @v6-sdn/Vlan35-all ip6 saddr != @v6-sdn/Vlan35-all-nomatch ip6 daddr @v6-sdn/Vlan36-all ip6 daddr != @v6-sdn/Vlan36-all-nomatch limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
ip saddr @v4-sdn/Vlan35-all ip saddr != @v4-sdn/Vlan35-all-nomatch ip daddr @v4-sdn/Vlan36-all ip daddr != @v4-sdn/Vlan36-all-nomatch accept
ip6 saddr @v6-sdn/Vlan35-all ip6 saddr != @v6-sdn/Vlan35-all-nomatch ip6 daddr @v6-sdn/Vlan36-all ip6 daddr != @v6-sdn/Vlan36-all-nomatch accept
limit rate 1/second log prefix ":0:7:bridge-Vlan36: ACCEPT: " group 0
accept

 

[shanreich]

How does your SDN configuration look like?

grep -r '' /etc/pve/sdn

How are you testing the firewall rules? ping? can you indicate the exact IPs / protocol / ports you are using to test the firewall connection?

 

[koniambo]

I'm testing via ping command (icmp) and various IP one from vlan 36 and one from vlan 35 on a machine which's in the vlan 36

Output of grep -r '' /etc/pve/sdn

/etc/pve/sdn/zones.cfg:vlan: vmbr0V
/etc/pve/sdn/zones.cfg: bridge vmbr0
/etc/pve/sdn/zones.cfg: ipam pve
/etc/pve/sdn/zones.cfg: mtu 9000
/etc/pve/sdn/zones.cfg:
/etc/pve/sdn/zones.cfg:vlan: Vlnvmbr0
/etc/pve/sdn/zones.cfg: bridge vmbr0
/etc/pve/sdn/zones.cfg: ipam pve
/etc/pve/sdn/zones.cfg: mtu 9000
/etc/pve/sdn/zones.cfg:
/etc/pve/sdn/zones.cfg:simple: Simple
/etc/pve/sdn/zones.cfg: ipam pve
/etc/pve/sdn/zones.cfg: mtu 9000
/etc/pve/sdn/zones.cfg:
/etc/pve/sdn/zones.cfg:simple: testsub
/etc/pve/sdn/zones.cfg: ipam pve
/etc/pve/sdn/zones.cfg: mtu 9000
/etc/pve/sdn/zones.cfg:
/etc/pve/sdn/zones.cfg:vxlan: vxlan36
/etc/pve/sdn/zones.cfg: peers 192.168.36.121,192.168.36.122
/etc/pve/sdn/zones.cfg: ipam pve
/etc/pve/sdn/zones.cfg: mtu 1370
/etc/pve/sdn/zones.cfg:
/etc/pve/sdn/.running-config:{"subnets":{"ids":{}},"version":43,"zones":{"ids":{"Vlnvmbr0":{"ipam":"pve","type":"vlan","mtu":9000,"bridge":"vmbr0"},"Simple":{"ipam":"pve","mtu":9000,"type":"simple"},"testsub":{"mtu":9000,"type":"simple","ipam":"pve"},"vmbr0V":{"ipam":"pve","type":"vlan","mtu":9000,"bridge":"vmbr0"},"vxlan36":{"peers":"192.168.36.121,192.168.36.122","type":"vxlan","mtu":1370,"ipam":"pve"}}},"vnets":{"ids":{"vxnet36":{"type":"vnet","vlanaware":1,"zone":"vxlan36","tag":36},"reseau35":{"type":"vnet","zone":"testsub"},"Vlan35":{"alias":"Vlan35_labtest","type":"vnet","vlanaware":1,"zone":"Vlnvmbr0","tag":35},"Vlan36":{"type":"vnet","vlanaware":1,"alias":"Vlan36_labtest","tag":36,"zone":"Vlnvmbr0"},"Vlan":{"vlanaware":1,"type":"vnet","alias":"Vlan_vmbr0","tag":199,"zone":"vmbr0V"}}},"controllers":{"ids":{}}}
/etc/pve/sdn/vnets.cfg:vnet: Vlan
/etc/pve/sdn/vnets.cfg: zone vmbr0V
/etc/pve/sdn/vnets.cfg: alias Vlan_vmbr0
/etc/pve/sdn/vnets.cfg: tag 199
/etc/pve/sdn/vnets.cfg: vlanaware 1
/etc/pve/sdn/vnets.cfg:
/etc/pve/sdn/vnets.cfg:vnet: Vlan35
/etc/pve/sdn/vnets.cfg: zone Vlnvmbr0
/etc/pve/sdn/vnets.cfg: alias Vlan35_labtest
/etc/pve/sdn/vnets.cfg: tag 35
/etc/pve/sdn/vnets.cfg: vlanaware 1
/etc/pve/sdn/vnets.cfg:
/etc/pve/sdn/vnets.cfg:vnet: Vlan36
/etc/pve/sdn/vnets.cfg: zone Vlnvmbr0
/etc/pve/sdn/vnets.cfg: alias Vlan36_labtest
/etc/pve/sdn/vnets.cfg: tag 36
/etc/pve/sdn/vnets.cfg: vlanaware 1
/etc/pve/sdn/vnets.cfg:
/etc/pve/sdn/vnets.cfg:vnet: reseau35
/etc/pve/sdn/vnets.cfg: zone testsub
/etc/pve/sdn/vnets.cfg:
/etc/pve/sdn/vnets.cfg:vnet: vxnet36
/etc/pve/sdn/vnets.cfg: zone vxlan36
/etc/pve/sdn/vnets.cfg: tag 36
/etc/pve/sdn/vnets.cfg: vlanaware 1
/etc/pve/sdn/vnets.cfg:
/etc/pve/sdn/firewall/Vlan35.fw:[OPTIONS]
/etc/pve/sdn/firewall/Vlan35.fw:
/etc/pve/sdn/firewall/Vlan35.fwolicy_forward: ACCEPT
/etc/pve/sdn/firewall/Vlan35.fw:enable: 1
/etc/pve/sdn/firewall/Vlan35.fw:log_level_forward: debug
/etc/pve/sdn/firewall/Vlan35.fw:
/etc/pve/sdn/firewall/Vlan35.fw:[RULES]
/etc/pve/sdn/firewall/Vlan35.fw:
/etc/pve/sdn/firewall/Vlan35.fw:FORWARD ACCEPT -source +sdn/Vlan-all -dest +sdn/Vlan-all -log nolog
/etc/pve/sdn/firewall/Vlan35.fw:
/etc/pve/sdn/firewall/Vlan36.fw:[OPTIONS]
/etc/pve/sdn/firewall/Vlan36.fw:
/etc/pve/sdn/firewall/Vlan36.fw:enable: 1
/etc/pve/sdn/firewall/Vlan36.fw:log_level_forward: debug
/etc/pve/sdn/firewall/Vlan36.fwolicy_forward: ACCEPT
/etc/pve/sdn/firewall/Vlan36.fw:
/etc/pve/sdn/firewall/Vlan36.fw:[RULES]
/etc/pve/sdn/firewall/Vlan36.fw:
/etc/pve/sdn/firewall/Vlan36.fw:FORWARD ACCEPT -source +sdn/Vlan36-all -dest +sdn/Vlan36-all -log debug
/etc/pve/sdn/firewall/Vlan36.fw:FORWARD ACCEPT -source +sdn/Vlan36-all -dest +sdn/Vlan35-all -log debug
/etc/pve/sdn/firewall/Vlan36.fw:FORWARD ACCEPT -source +sdn/Vlan35-all -dest +sdn/Vlan35-all -log debug
/etc/pve/sdn/firewall/Vlan36.fw:FORWARD ACCEPT -source +sdn/Vlan35-all -dest +sdn/Vlan36-all -log debug
/etc/pve/sdn/firewall/Vlan36.fw:
/etc/pve/sdn/firewall/vxnet36.fw:[OPTIONS]
/etc/pve/sdn/firewall/vxnet36.fw:
/etc/pve/sdn/firewall/vxnet36.fw:enable: 1
/etc/pve/sdn/firewall/vxnet36.fw:
/etc/pve/sdn/firewall/Vlan.fw:[OPTIONS]
/etc/pve/sdn/firewall/Vlan.fw:
/etc/pve/sdn/firewall/Vlan.fwolicy_forward: ACCEPT
/etc/pve/sdn/firewall/Vlan.fw:enable: 1
/etc/pve/sdn/firewall/Vlan.fw:log_level_forward: debug
/etc/pve/sdn/firewall/Vlan.fw:
/etc/pve/sdn/pve-ipam-state.json:{"zones":{"testsub":{"subnets":{}},"Vlnvmbr0":{"subnets":{}},"Simple":{"subnets":{}}}}

 

[shanreich]

I'm testing via ping command (icmp) and various IP one from vlan 36 and one from vlan 35 on a machine which's in the vlan 36

I'm not sure I understand correctly? You configure an IP from vlan35 on a machine with its interface on vlan36? And on another machine an IP from vlan36 with its interface on vlan36?

In any case: You do not have any subnets configured for your vnets, the firewall takes the information about IP ranges from the subnet configuration of the VNets, so those IPsets you are using are empty which causes the firewall rules to not match.

Also note: the VNet-level firewall is only for traffic inside the same VNet, so from all traffic that is crossing the bridge [1] (e.g. vlan36 <-> vlan 36). If you have traffic between different VLANs, then you would need to create the firewall rules on your router that routes between the different VLANs (so, outside of PVE usually).

Does pinging between two machines on the same vnet (e.g. Vlan36) work with the rules above?
If not, could you post an example output of nft monitor trace using the commands from the documentation? [2]. How does the VM configuration look like (qm config <vmid>) and how does the network configuration inside the VMs look like (i.e. ip a / ip r)? What is the exact output of the ping command?

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_zones
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pve_firewall_nft_helpful_commands