Problem firewall dropping packet

andaga · Sep 4, 2024

Hello everyone!

I have a problem that I have been working on for 4 days now and that is driving me crazy! I just figured out where the problem was and frankly I am very surprised..... It is the ProxMox firewall that is messing up my PBX phone system!

In summary:

When I make a call from my WebRTC software it turns out that ProxMox Firewall drops some packets so consequently my pfsense never receives the packet and neither does my trunk provider which prevents the communication from being established.

What I have noticed is that even though the firewall at the VM level is DISABLED! As long as the firewall is active at the datacenter level it blocks my calls because a packet is dropped! If I disable the firewall at the datacenter level, everything works normally!

How can I say that if the VM firewall is not active, basically the DC firewall shuts its mouth and lets the traffic pass normally?

Here are some screenshots of the ProxMox Firewall config!

THANK YOU in advance for your help! It's relatively urgent.

shanreich · Sep 4, 2024

How does the network configuration of the VM look like? Is it NATed?

Can you post the output of the following commands:

Code:

qm config <vmid>
cat /etc/network/interfaces

andaga · Sep 4, 2024

Code:

root@FRI-KVM-50:~# qm config 355
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cipassword: **********
ciuser: root
cores: 4
cpu: x86-64-v2-AES
efidisk0: PVE-CH-FRI:355/vm-355-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
ide0: PVE-CH-FRI:355/vm-355-cloudinit.qcow2,media=cdrom
ide2: none,media=cdrom
ipconfig0: ip=171.33.24xxxx/24,gw=171.x33.2xxxx.1,ip6=2a12xxxxx:1::15/64,gw6=2a12:44c0:0xxxxxx:1::1
memory: 8192
meta: creation-qemu=8.1.2,ctime=1702603565
name: pbx.mxxxxxxxxct.ch
nameserver: 8.8.8.8 9.9.9.9 1.1.1.1
net0: virtio=BC:2xxxxx4:72,bridge=vmbr1,tag=2xx0
numa: 0
ostype: l26
scsi0: PVE-CH-FRI:355/vm-355-disk-1.qcow2,iothread=1,size=100G
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=da92d40xxxxxc5-9ceecxxxxxxxxx350
sockets: 1
vmgenid: 6c2d575d-abbdxxxxxxe4868428b
root@FRI-KVM-50:~#

Code:

root@FRI-KVM-50:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp2s0f0np0
iface enp2s0f0np0 inet manual

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp2s0f1np1
iface enp2s0f1np1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp2s0f0np0 enp2s0f1np1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp2s0f0np0

auto bond1
iface bond1 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode active-backup
        bond-primary eno1

auto vmbr0
iface vmbr0 inet static
        address 172.16.20.50/20
        gateway 172.16.16.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

iface vmbr0 inet6 static
        address fd00:1234:5678::10/48
        gateway fd00:1234:5678::1

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

source /etc/network/interfaces.d/*
root@FRI-KVM-50:~#

andaga · Sep 4, 2024

Hello! No NAT! The VM have directly a public IP assigned IPV4 and IPv6 behind a VLAN.

shanreich · Sep 4, 2024

Are you connecting via IPv4 or IPv6, or are both broken?
Where is the server located that you are trying to connect to?

andaga · Sep 4, 2024

We are trying to connect over ipv4
The server is located in switzerland and trying to connect to another server in switzerland!

andaga · Sep 4, 2024

Also another comment!
What is very strange with proxmox firewall.. when we call with softphone or phone like yealink or other we don't have any issue... Only when we want call with the webrtc of the pbx proxmox is dropping packet with firewall active at datacenter level

andaga · Sep 4, 2024

Any idea? Maybe a bug into proxmox firewall ?

andaga · Sep 4, 2024

Someone have an idea?

andaga · Sep 5, 2024

shanreich said:
Are you connecting via IPv4 or IPv6, or are both broken?
Where is the server located that you are trying to connect to?

Hello! Do you have some news for me?

shanreich · Sep 5, 2024

Could you post the output of iptables-save when the firewall is running?

I'll have to admit I only have very rudimentary knowledge of WebRTC, so I cannot 100% tell how the protocol operates. Could you post a tcpdump of a failed WebRTC call - captured on the interface of the PBX? (tap355i0) Maybe I can find something there.

You said, you have a pfSense running - is it running inside the cluster or is it external to the cluster?

andaga · Sep 5, 2024

My pfsense is hardware firewall so external of the cluster! The pfsense never receive the packet it's really blocked by the proxmox firewall!

Here is the pcap file during a call from proxmox and the pcap file from our pfsense.

Code:

root@FRI-KVM-50:~# iptables-save
# Generated by iptables-save v1.8.9 on Thu Sep  5 11:22:58 2024
*raw
:PREROUTING ACCEPT [297985495:102655693272]
:OUTPUT ACCEPT [281069468:236416736858]
COMMIT
# Completed on Thu Sep  5 11:22:58 2024
# Generated by iptables-save v1.8.9 on Thu Sep  5 11:22:58 2024
*filter
:INPUT ACCEPT [33650:3650695]
:FORWARD ACCEPT [73541:5720575]
:OUTPUT ACCEPT [28195:1758601]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 80 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 443 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 6556 -j RETURN
-A PVEFW-HOST-IN -p icmp -m icmp --icmp-type 8 -j RETURN
-A PVEFW-HOST-IN -p udp -m udp --dport 4789 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.51/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.52/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.53/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.54/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.55/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.56/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 172.16.20.57/32 -d 172.16.20.50/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:ux/X+41+ig8tTYo/cZNGTyUfY/0"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d 172.16.16.0/20 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 172.16.16.0/20 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 172.16.16.0/20 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 172.16.16.0/20 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.51/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.52/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.53/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.54/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.55/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.56/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -s 172.16.20.50/32 -d 172.16.20.57/32 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:GnTyacMl5mUEJzVSyLDp6YrxEZY"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Thu Sep  5 11:22:58 2024
root@FRI-KVM-50:~#

andaga · Sep 5, 2024

shanreich said:
I'll have to admit I only have very rudimentary knowledge of WebRTC, so I cannot 100% tell how the protocol operates. Could you post a tcpdump of a failed WebRTC call - captured on the interface of the PBX? (tap355i0) Maybe I can find something there.

Your knowledge is not a problem! The problem is proxmox firewall.

Firewall eneable at DC level = ok
FIrewall enable at host level = ok
FIrewall disable at vm level= NO OK!

The problem is WHY Firewall enabled at the DC level block something from a VM where the status is disabled!?

shanreich · Sep 5, 2024

andaga said:
The problem is WHY Firewall enabled at the DC level block something from a VM where the status is disabled!?

There could be a few explanations, the most likely is that enabling the firewall enables conntrack on the host, which would also affect VMs (since their connections get conntracked as well then and there are rules that drop traffic with invalid conntrack state).

andaga · Sep 5, 2024

shanreich said:
There could be a few explanations, the most likely is that enabling the firewall enables conntrack on the host, which would also affect VMs (since their connections get conntracked as well then and there are rules that drop traffic with invalid conntrack state).

Ok thank you for this clarification... but how to can fix my issue? Because we absolutly need the firewall for all other vm! But it block the pbx! So we are on the sh....itttt

shanreich · Sep 5, 2024

Looks like the issue is with the MTU of the responses, it is bigger than 1500 and because of this you're probably running into this bug [1] of the firewall. You will either need to increase the MTU of the respective interfaces (which can still lead to problems if some packets are larger than the MTU you provided). Or switch to the experimental nftables firewall (note that it is still tech-preview and might contain other bugs), where this bug is fixed.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=4158

andaga · Sep 5, 2024

Thank you so it's really a bug lol! I was crazy with this issue...

How to move to the new firewall experimental? When it will be official?

shanreich · Sep 5, 2024

andaga said:
How to move to the new firewall experimental? When it will be official?

You should be able to enable it in the Host Firewall options. Make sure to have the newest package installed before you do, since it fixes a few bugs with the initial release, particularly when using VLANs.

We currently have no roadmap when the firewall will replace the old one, but it almost certainly won't be before the next major release (not saying that it will be the default at the next major release).

andaga · Sep 5, 2024

I have change the settings to nftables (tech preview) to yes but problem still the same

I have changed the firewall mode into all nodes of the cluster.

I have run: pve-firewall stop
pve-firewall start

and also pve-firewall restart

but the problem remain the same!

shanreich · Sep 5, 2024

Do you have the firewall enabled at the NIC in the VM configuration? You need to restart the VM after enabling the new firewall to make sure it does not create an additional firewall bridge.

The service for the new firewall is called proxmox-firewall, so you will need to check the status with the following command

Code:

systemctl status proxmox-firewall

Problem firewall dropping packet

Member

Attachments

Proxmox Staff Member

Member

Member

Proxmox Staff Member

Member

Member

Member

Member

Member

Proxmox Staff Member

Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member