Firewall multicast rules

Z

zeuxprox

Renowned Member
Proxmox Subscriber
Dec 10, 2014
92
5
73
Hi,

I created a cluster of 4 nodes, now I would like to know which rule I have to add, in the firewall gui, to permit multicast traffic on the management subnet (192.168.15.0/24 , iface vmbr0)...

Thank you very much
 
Hi,

I added the following rules to the firewall gui:

Rules n.1:

Direction: IN
Action: ACCEPT
Source: left blank
Destination: left blank
Macro: left blank
Protocol: udp
Source Port: left blank
Destination Port: 5404:5405


Rules n.2:
Direction: IN
Action: ACCEPT
Source: left blank
Destination: left blank
Macro: left blank
Protocol: igmp
Source Port: left blank
Destination Port: left blank

The problem is that the Firewall blocks the multicast packets and therefore corosynd doesn't works...

Help me, please...

Thank you very much
 
Hi,
by default PVE's firewall rules allow the multicast communication for the hosts in order for corosync to work. Please post your `iptables-save` output and the host config `cat /etc/pve/nodes/<nodename>/host.fw` if that's not the case.
 
Hi,

/etc/pve/nodes/prx1/host.fw

[OPTIONS]

enable: 1

[RULES]

GROUP managementipmi # Management IPMI to ManagementVM
GROUP ceph_private -i ceph23 # Ceph Private Subnet OK
GROUP ceph_public -i ceph22 # Ceph Public OK
GROUP migrationvm -i migr21 # MigrationVM Access
GROUP management -i mgmt20 # Management Access


iptables-save:

*filter
:INPUT ACCEPT [2289:410696]
:FORWARD ACCEPT [26:1456]
:OUTPUT ACCEPT [2167:364667]
:GROUP-ceph_private-IN - [0:0]
:GROUP-ceph_private-OUT - [0:0]
:GROUP-ceph_public-IN - [0:0]
:GROUP-ceph_public-OUT - [0:0]
:GROUP-managementipmi-IN - [0:0]
:GROUP-managementipmi-OUT - [0:0]
:GROUP-management-IN - [0:0]
:GROUP-management-OUT - [0:0]
:GROUP-migrationvm-IN - [0:0]
:GROUP-migrationvm-OUT - [0:0]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A GROUP-ceph_private-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-ceph_private-IN -s 192.168.23.0/24 -d 192.168.23.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-ceph_private-IN -m comment --comment "PVESIG:iSPPdpQmVwyqbZY0hq58t3Ejoo0"
-A GROUP-ceph_private-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-ceph_private-OUT -m comment --comment "PVESIG:x6h0tdlLIz6JEtq7KPVHGjikoes"
-A GROUP-ceph_public-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-ceph_public-IN -s 192.168.22.0/24 -d 192.168.22.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-ceph_public-IN -m comment --comment "PVESIG:tO331haEjXJlEdauWJexVVYQ67Y"
-A GROUP-ceph_public-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-ceph_public-OUT -m comment --comment "PVESIG:2d/rItjFb62GE+JtP23AjtxX/58"
-A GROUP-managementipmi-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-managementipmi-IN -s 192.168.17.0/24 -d 192.168.15.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-managementipmi-IN -m comment --comment "PVESIG:L5zZZWpxd+hvwu1HEGXNjgRcP8I"
-A GROUP-managementipmi-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-managementipmi-OUT -m comment --comment "PVESIG:48Lp7+2vdw28Dt+15L8aY+EOhIc"
-A GROUP-management-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-management-IN -p udp -m udp --dport 5404:5405 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -p igmp -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -d 192.168.15.0/24 -p tcp -m set --match-set PVEFW-0-subnets_allow-v4 src -m tcp --dport 2228 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -d 192.168.15.0/24 -p tcp -m set --match-set PVEFW-0-subnets_allow-v4 src -m tcp --dport 3128 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -d 192.168.15.0/24 -p tcp -m set --match-set PVEFW-0-subnets_allow-v4 src -m tcp --dport 5900:5999 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -d 192.168.15.0/24 -p icmp -m set --match-set PVEFW-0-subnets_allow-v4 src -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -d 192.168.15.0/24 -p tcp -m set --match-set PVEFW-0-subnets_allow-v4 src -m tcp --dport 8006 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -s 192.168.15.0/24 -d 192.168.15.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -s 127.0.0.0/24 -d 127.0.0.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-management-IN -m comment --comment "PVESIG:hFR6qs1xpOSf63sVruTrq/bJZH0"
-A GROUP-management-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-management-OUT -m comment --comment "PVESIG:507jzQvmkj7pyZYZEiI0n/PJt4A"
-A GROUP-migrationvm-IN -j MARK --set-xmark 0x0/0x80000000
-A GROUP-migrationvm-IN -s 10.18.21.0/24 -d 10.18.21.0/24 -g PVEFW-SET-ACCEPT-MARK
-A GROUP-migrationvm-IN -m comment --comment "PVESIG:evuR6fzt3gsd+ZPthdmNA4RKx/A"
-A GROUP-migrationvm-OUT -j MARK --set-xmark 0x0/0x80000000
-A GROUP-migrationvm-OUT -m comment --comment "PVESIG:a7a9nsI9iHOfikpvrGPaRuL1SiQ"
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -j GROUP-managementipmi-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i ceph23 -j GROUP-ceph_private-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i ceph22 -j GROUP-ceph_public-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i migr21 -j GROUP-migrationvm-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -i mgmt20 -j GROUP-management-IN
-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -s 192.168.15.0/24 -d 192.168.15.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.15.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:fPJC4UQjg0M3fPM72ECqDEX9SZ8"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -j GROUP-managementipmi-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o ceph23 -j GROUP-ceph_private-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o ceph22 -j GROUP-ceph_public-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o migr21 -j GROUP-migrationvm-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -o mgmt20 -j GROUP-management-OUT
-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:a7KMZMBuoFLMXLN/iSj4gWj3WdE"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tc-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT

Thank you very much
 
Code: 
-A PVEFW-HOST-IN -s 192.168.15.0/24 -d 192.168.15.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.15.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
Code: 
-A PVEFW-HOST-OUT -d 192.168.15.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
These rules you have already set should make sure multicast for corosync traffic is allowed. Did you test multicast with omping? See here for more information https://pve.proxmox.com/wiki/Multicast_notes
 
Hi,
yes I tested multicast with omping and % of multicast and unicast packets loss is 0% (with firewall disabled). My switches are cisco nexus 3064 and the configuration is (vlan 15 is the management vlan):

vlan configuration 15
ip igmp snooping querier 192.168.15.253
ip igmp snooping fast-leave

Omping test excuted on the first (192.168.15.51) and second (192.168.15.52) nodes with the firewall disabled:

omping -c 10000 -i 0.001 -F -q 192.168.15.51 192.168.15.52

192.168.15.52 : joined (S,G) = (*, 232.43.211.234), pinging
192.168.15.52 : waiting for response msg
192.168.15.52 : server told us to stop

192.168.15.52 : unicast, xmt/rcv/%loss = 9461/9461/0%, min/avg/max/std-dev = 0.024/0.077/0.161/0.016
192.168.15.52 : multicast, xmt/rcv/%loss = 9461/9461/0%, min/avg/max/std-dev = 0.025/0.061/0.384/0.022


If I enable the firewall 100% of multicast packets are lost...

Thank you
 
Okay, you can try to see where the packets get dropped by turning on debugging output for the Firewall rules.
 
I have the exact same issue. Once PVE firewall is enabled, multicast not going through, despite what rule I enable.
 
Multicast was working for me until I ran a package update yesterday and rebooted one of the nodes in a two-node cluster (start of a bigger cluster) and when it came back up I couldn't get it joined back into the cluster until I switched to unicast in corosync.conf.

Still investigating, but it's weird.

Switching firewall logging to 'debug' yields no log entries when I run an `omping` test, which I would expect if it's the fw blocking multicast, right?
 
Last edited:
Okay, so multicast works, but only if I manually specify the multicast address 224.0.0.1

Code: 
omping -m 224.0.0.1 node1 node2

But what I get when I run
Code: 
corosync-cmapctl -g totem.interface.0.mcastaddr

is
Code: 
totem.interface.0.mcastaddr (str) = 239.192.23.137



EDIT:
Fix was simple, I just added a `mcastaddr` directive to /etc/corosync/corosync.conf like so:
Code: 
  interface {
    bindnetaddr: 10.10.10.28
    ringnumber: 0
    mcastaddr: 224.0.0.1
  }

Still not sure why it broke, but at least it works again. Hope this helps someone else in the future.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom