[SOLVED] Cluster loses quorum when activating Firewall

HE_Cole

Member
Oct 25, 2018
45
0
6
28
Miami, FL
Hello Everyone.

Sorry i have had a lot of questions on here lately and i really do appreciate the help.

I finally gott my PVE firewall to enable and its working but once i enable the firewall all nodes lose quorum but from the pve GUI i can still access all nodes in the cluster and ceph and remote access are fine too. Yet as soon as i enable the firewall in the pve gui all nodes turn red and lose quorum but every thing else on each node works ceph and ssh are fine its just quorm stop working.

I think the issue is multicast us being blocked BUT i created a rule in the gui to allow in/out all from the multicast ip range 224.0.0.0/4 and i created a allow in/out for the corosync subnet 172.16.16.0/25.

With the firewall on i can ping from and to any node using the corosync ip's for each node so corosync traffic is working yet no quorum is not working when the firewall is ON.

My Corosync is on a sepreate interface and subnet.

Any ideas?

Here is my pve-firewall compile

Code:
root@he-s01-r01-pve01:~# pve-firewall compile
ipset cmdlist:
create PVEFW-0-management-v4 (VELRN4U0uxie5kc56f6CC6Kraxw)
        create PVEFW-0-management-v4 hash:net family inet hashsize 64 maxelem 64
        add PVEFW-0-management-v4 23.136.0.0/28
create PVEFW-0-management-v6 (H5WO/Pkuyz4e7OLB2uiMpG0Bsn0)
        create PVEFW-0-management-v6 hash:net family inet6 hashsize 64 maxelem 64

iptables cmdlist:
create GROUP-allow_ceph-IN (b+RbIpoR2T1ZTdhmr5VXndsFgkw)
        -A GROUP-allow_ceph-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ceph-IN -s 10.10.11.0/24 -d 10.10.11.0/24 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_ceph-IN -s 10.10.12.0/24 -d 10.10.12.0/24 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ceph-OUT (gf+4SboZzstjCs+MHLAWKNHELG4)
        -A GROUP-allow_ceph-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ceph-OUT -s 10.10.11.0/24 -d 10.10.11.0/24 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_ceph-OUT -s 10.10.12.0/24 -d 10.10.12.0/24 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_gui-IN (wXO5F1XiI0syHY+X8D6E6uR7PyY)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_gui-IN -s 0.0.0.0/0 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 8006 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-IN (aRLtGRcAJTiHifd3QiBxLaSXtaY)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_local-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_local-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_local-OUT (oiiNF92sRuXzWsWG7GY/E+0j81A)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_local-OUT -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_local-OUT -s 23.136.0.0/28 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ssh-IN (xB/wEFCnbU3Y3iz2Fz5PIVYTGAA)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ssh-IN -s 0.0.0.0/0 -d 23.136.0.0/28 -p tcp --dport 22 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-IN (SPe17bcBlUraKkvBwPuQmOw/evQ)
        -A GROUP-allow_sync-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_sync-IN -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_sync-IN -s 172.16.16.0/25 -d 172.16.16.0/25 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_sync-OUT (3um/xpUxqrxywB1jlQMcRNOff/I)
        -A GROUP-allow_sync-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_sync-OUT -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_sync-OUT -s 172.16.16.0/25 -d 172.16.16.0/25 -g PVEFW-SET-ACCEPT-MARK
create PVEFW-Drop (WDy2wbFe7jNYEyoO3QhUELZ4mIQ)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Drop -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (NyjHNAtFbkH7WGLamPpdVnxHy4w)
        -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
        -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (Ijl7/xz0DD7LF91MlLCz0ybZBE0)
        -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
create PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-HOST-IN (0zmDTqXdwIEfl6Ig2HSLVDvdFJc)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno2 -j GROUP-allow_sync-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr2 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr1 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/28 -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (2zC4RjJsBRdAGMBg/jHpy7VHOZk)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno2 -j GROUP-allow_sync-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr2 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr1 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (CZJnIN6rAdpu+ej59QPr9+laMUo)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Reject -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
create PVEFW-reject (Jlkrtle1mDdtxDeI9QaDSL++Npc)
        -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-reject -s 224.0.0.0/4 -j DROP
        -A PVEFW-reject -p icmp -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
        -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
        -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
        -A PVEFW-reject  -j REJECT --reject-with icmp-host-prohibited
create PVEFW-smurflog (2gfT1VMkfr0JL6OccRXTGXo+1qk)
        -A PVEFW-smurflog  -j DROP
create PVEFW-smurfs (HssVe5QCBXd5mc9kC88749+7fag)
        -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
        -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
        -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ip6tables cmdlist:
create GROUP-allow_ceph-IN (7THbUgKtqSW7VYmAamnnyUhaink)
        -A GROUP-allow_ceph-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ceph-OUT (g1sqjMPDECYMWaWoi1LVf1ym+dw)
        -A GROUP-allow_ceph-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_gui-IN (C2ZOZ61Y9OGj0gEkT8Dpm82CpWI)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-IN (4TDlQ1Iq1o6UBZ1EKmiMphvG9fU)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ssh-IN (qUiol7H1++Cjk2TCBJsMuXzJt+8)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-IN (3QdQE7vXIeuhLt/bb+tCKhOVcgY)
        -A GROUP-allow_sync-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-OUT (Uyb+r1bfSlMI9lhniKFzKnxyV9s)
        -A GROUP-allow_sync-OUT -j MARK --set-mark 0x00000000/0x80000000
create PVEFW-Drop (Jb79Uw7z1vZglIcV7QXA5uY/nbk)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (8Krk5Nh8pDZOOc7BQAbM6PlyFSU)
        -A PVEFW-DropBroadcast -d ff00::/8 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-HOST-IN (LnkexloL8+lzKK3dH4/gGLuWY0I)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-advertisement -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno2 -j GROUP-allow_sync-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr2 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr1 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (rB/xLO72g/pHiCL0u3sWqn5WwLE)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno2 -j GROUP-allow_sync-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr2 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr1 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (aL1nrxJk/u3XmTb3Am2eaM/3yCM)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
create PVEFW-reject (TeZhczhc17LK2pqE7UkGmRMJLNU)
        -A PVEFW-reject -p icmpv6 -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
        -A PVEFW-FORWARD -p IPv4 -j ACCEPT
        -A PVEFW-FORWARD -p IPv6 -j ACCEPT
        -A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore FORWARD (zuQi5YOvmMWiM9zohnQw/qWemOA)
ignore INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
detected changes
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,592
375
103
Hi,
My Corosync is on a sepreate interface and subnet.
that's fine so you can disable the firewall on this interfaces if they are isolated.
Or allow all traffic from the nodes to the nodes on this interface.
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,592
375
103
You can choose the interface on the rule.
So not choose the interface what corosync useses.
 
  • Like
Reactions: HE_Cole

HE_Cole

Member
Oct 25, 2018
45
0
6
28
Miami, FL
Hey wolfgang!

I tried just not adding a rule to the corosync and ceph interfaces but my cluster still drops quroum when i enable the firewall. I only have rules on my vmbr0 which is my WAN interface and did not add rules to other interfaces and yet still enable firewall still brakes ceph and corosync.

With the firewall ON I can still access each node from the pve gui just fine but all nodes are red and have a X and clicking Ceph in the gui under a node just gives a 500 error with the firewall on? And one i turn the firewall on i have to manually turn it off on each node as if i click under data center to turn the firewall off i get error " unable to open file '/etc/pve/firewall/cluster.fw.tmp.2912332' - Permission denied (500)
and i have to go to each node and run pve-firewall stop THEN i can disable to firewall in Data center without error.

Any ideas still?
 

HE_Cole

Member
Oct 25, 2018
45
0
6
28
Miami, FL
I dont mean to be bugging the communty but i would like some help on this firewall issue i am sure its simple.

I did what wolfgang and i did not any rules for my corosync and ceph interfaces, With the firewall on i can ping on them fine and that part works.

But i still loose quoram and nodes cant ping each other when i turn the firewall on under datacenter. I even have a rule for ALLOW in 0.0.0.0/0 to 0.0.0.0/0 and a ALLOW OUT in postion 0 in the firewall and still my node to node gets blocked.

Here is my pve-firewall

Code:
root@he-s01-r01-pve01:~# pve-firewall compile
ipset cmdlist:
exists PVEFW-0-management-v4 (LRAv4c0S4qV/TUNkSvLtGKJQpJE)
        create PVEFW-0-management-v4 hash:net family inet hashsize 64 maxelem 64
        add PVEFW-0-management-v4 23.136.0.0/24
exists PVEFW-0-management-v6 (H5WO/Pkuyz4e7OLB2uiMpG0Bsn0)
        create PVEFW-0-management-v6 hash:net family inet6 hashsize 64 maxelem 64

iptables cmdlist:
exists GROUP-allow-any-any-IN (CmRRPmWWGjQB0wNXO+7sGTZqqyI)
        -A GROUP-allow-any-any-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow-any-any-IN -s 0.0.0.0/0 -d 0.0.0.0/0 -g PVEFW-SET-ACCEPT-MARK
exists GROUP-allow-any-any-OUT (3SLSpkCfprkcOPMdgrTtP4aXG+g)
        -A GROUP-allow-any-any-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow-any-any-OUT -s 0.0.0.0/0 -d 0.0.0.0/0 -g PVEFW-SET-ACCEPT-MARK
exists GROUP-allow_gui-IN (zNslOg51fEKOPXIk97ncrr1KmUY)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_gui-IN -s 0.0.0.0/0 -d 23.136.0.0/24 -p tcp --sport 1:65535 --dport 8006 -g PVEFW-SET-ACCEPT-MARK
exists GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_local-IN (8KsLa7QApT6AiW4s1PHMXkre9Uc)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_local-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_local-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
exists GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_ssh-IN (abnjYentOoN3Pb779drx14aOswg)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ssh-IN -s 0.0.0.0/0 -d 23.136.0.0/24 -p tcp --dport 22 -g PVEFW-SET-ACCEPT-MARK
exists GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
exists PVEFW-Drop (WDy2wbFe7jNYEyoO3QhUELZ4mIQ)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Drop -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
exists PVEFW-DropBroadcast (NyjHNAtFbkH7WGLamPpdVnxHy4w)
        -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
        -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
exists PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-IN (Ijl7/xz0DD7LF91MlLCz0ybZBE0)
        -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
exists PVEFW-HOST-IN (/OOHUEhE0qH9Amh6YKkiDcoF+K8)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow-any-any-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/24 -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
exists PVEFW-HOST-OUT (dyEWVNmhC2MWgSHnilAoV53DDhU)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow-any-any-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
exists PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
exists PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
exists PVEFW-Reject (CZJnIN6rAdpu+ej59QPr9+laMUo)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Reject -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
exists PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
exists PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
exists PVEFW-reject (Jlkrtle1mDdtxDeI9QaDSL++Npc)
        -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-reject -s 224.0.0.0/4 -j DROP
        -A PVEFW-reject -p icmp -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
        -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
        -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
        -A PVEFW-reject  -j REJECT --reject-with icmp-host-prohibited
exists PVEFW-smurflog (2gfT1VMkfr0JL6OccRXTGXo+1qk)
        -A PVEFW-smurflog  -j DROP
exists PVEFW-smurfs (HssVe5QCBXd5mc9kC88749+7fag)
        -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
        -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
        -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
exists PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ip6tables cmdlist:
exists GROUP-allow-any-any-IN (wOWApKvEMrqx/FoWp4UQoIhRklQ)
        -A GROUP-allow-any-any-IN -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow-any-any-OUT (wbpUG4+DVKK9sFIDvmWl7xXic28)
        -A GROUP-allow-any-any-OUT -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_gui-IN (C2ZOZ61Y9OGj0gEkT8Dpm82CpWI)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_local-IN (4TDlQ1Iq1o6UBZ1EKmiMphvG9fU)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_ssh-IN (qUiol7H1++Cjk2TCBJsMuXzJt+8)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
exists GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
exists PVEFW-Drop (Jb79Uw7z1vZglIcV7QXA5uY/nbk)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
exists PVEFW-DropBroadcast (8Krk5Nh8pDZOOc7BQAbM6PlyFSU)
        -A PVEFW-DropBroadcast -d ff00::/8 -j DROP
exists PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-IN (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
exists PVEFW-HOST-IN (1AdmChTabG9C/+Mds0CMQd+F8HA)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-advertisement -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow-any-any-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
exists PVEFW-HOST-OUT (NsCZQiShTEOT4oQqQSeIIGYwBcU)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow-any-any-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
exists PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
exists PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
exists PVEFW-Reject (aL1nrxJk/u3XmTb3Am2eaM/3yCM)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
exists PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
exists PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
exists PVEFW-reject (TeZhczhc17LK2pqE7UkGmRMJLNU)
        -A PVEFW-reject -p icmpv6 -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
exists PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
        -A PVEFW-FORWARD -p IPv4 -j ACCEPT
        -A PVEFW-FORWARD -p IPv6 -j ACCEPT
        -A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore FORWARD (zuQi5YOvmMWiM9zohnQw/qWemOA)
ignore INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
no changes
root@he-s01-r01-pve01:~#

I tryed adding the rule on the interface level and the bridge level and both same issue.


Thanks..
 

wolfgang

Proxmox Staff Member
Staff member
Oct 1, 2014
5,592
375
103
To debug your Firewall a network config is needed.
send the output of this two commands

Code:
iptables -L
ip a
 

spirit

Famous Member
Apr 2, 2010
3,917
210
83
www.odiso.com
for multicast, source and destination network are not the same


>> -A GROUP-allow_sync-IN -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
>> -A GROUP-allow_sync-OUT -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK

the source is the real ip address of host from were corosync send packets.


(you can verify that with a tcpdump)
 

HE_Cole

Member
Oct 25, 2018
45
0
6
28
Miami, FL
Issue resolved.

SOLUTION:
I just set the Default INPUT to ALLOW in PVE firewall and then added a BLOCK-ALL 4v/6vTCP/UDP rule at the end of my Allow list and everything works fine on my cluster now and proper traffic is getting blocked and allowed traffic is allowed.

Thanks so much everyone for the help.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!