[SOLVED] Cluster loses quorum when activating Firewall

HE_Cole · Dec 7, 2018

Hello Everyone.

Sorry i have had a lot of questions on here lately and i really do appreciate the help.

I finally gott my PVE firewall to enable and its working but once i enable the firewall all nodes lose quorum but from the pve GUI i can still access all nodes in the cluster and ceph and remote access are fine too. Yet as soon as i enable the firewall in the pve gui all nodes turn red and lose quorum but every thing else on each node works ceph and ssh are fine its just quorm stop working.

I think the issue is multicast us being blocked BUT i created a rule in the gui to allow in/out all from the multicast ip range 224.0.0.0/4 and i created a allow in/out for the corosync subnet 172.16.16.0/25.

With the firewall on i can ping from and to any node using the corosync ip's for each node so corosync traffic is working yet no quorum is not working when the firewall is ON.

My Corosync is on a sepreate interface and subnet.

Any ideas?

Here is my pve-firewall compile

Code:

root@he-s01-r01-pve01:~# pve-firewall compile
ipset cmdlist:
create PVEFW-0-management-v4 (VELRN4U0uxie5kc56f6CC6Kraxw)
        create PVEFW-0-management-v4 hash:net family inet hashsize 64 maxelem 64
        add PVEFW-0-management-v4 23.136.0.0/28
create PVEFW-0-management-v6 (H5WO/Pkuyz4e7OLB2uiMpG0Bsn0)
        create PVEFW-0-management-v6 hash:net family inet6 hashsize 64 maxelem 64

iptables cmdlist:
create GROUP-allow_ceph-IN (b+RbIpoR2T1ZTdhmr5VXndsFgkw)
        -A GROUP-allow_ceph-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ceph-IN -s 10.10.11.0/24 -d 10.10.11.0/24 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_ceph-IN -s 10.10.12.0/24 -d 10.10.12.0/24 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ceph-OUT (gf+4SboZzstjCs+MHLAWKNHELG4)
        -A GROUP-allow_ceph-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ceph-OUT -s 10.10.11.0/24 -d 10.10.11.0/24 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_ceph-OUT -s 10.10.12.0/24 -d 10.10.12.0/24 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_gui-IN (wXO5F1XiI0syHY+X8D6E6uR7PyY)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_gui-IN -s 0.0.0.0/0 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 8006 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-IN (aRLtGRcAJTiHifd3QiBxLaSXtaY)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_local-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_local-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_local-OUT (oiiNF92sRuXzWsWG7GY/E+0j81A)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_local-OUT -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_local-OUT -s 23.136.0.0/28 -d 23.136.0.0/28 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ssh-IN (xB/wEFCnbU3Y3iz2Fz5PIVYTGAA)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_ssh-IN -s 0.0.0.0/0 -d 23.136.0.0/28 -p tcp --dport 22 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-IN (SPe17bcBlUraKkvBwPuQmOw/evQ)
        -A GROUP-allow_sync-IN -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_sync-IN -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_sync-IN -s 172.16.16.0/25 -d 172.16.16.0/25 -g PVEFW-SET-ACCEPT-MARK
create GROUP-allow_sync-OUT (3um/xpUxqrxywB1jlQMcRNOff/I)
        -A GROUP-allow_sync-OUT -j MARK --set-mark 0x00000000/0x80000000
        -A GROUP-allow_sync-OUT -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
        -A GROUP-allow_sync-OUT -s 172.16.16.0/25 -d 172.16.16.0/25 -g PVEFW-SET-ACCEPT-MARK
create PVEFW-Drop (WDy2wbFe7jNYEyoO3QhUELZ4mIQ)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Drop -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (NyjHNAtFbkH7WGLamPpdVnxHy4w)
        -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
        -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
        -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (Ijl7/xz0DD7LF91MlLCz0ybZBE0)
        -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
create PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-HOST-IN (0zmDTqXdwIEfl6Ig2HSLVDvdFJc)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno2 -j GROUP-allow_sync-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr2 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr1 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/28 -d 23.136.0.0/28 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -s 23.136.0.0/28 -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (2zC4RjJsBRdAGMBg/jHpy7VHOZk)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno2 -j GROUP-allow_sync-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr2 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr1 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-OUT -d 23.136.0.0/28 -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (CZJnIN6rAdpu+ej59QPr9+laMUo)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
        -A PVEFW-Reject -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
create PVEFW-reject (Jlkrtle1mDdtxDeI9QaDSL++Npc)
        -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
        -A PVEFW-reject -s 224.0.0.0/4 -j DROP
        -A PVEFW-reject -p icmp -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
        -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
        -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
        -A PVEFW-reject  -j REJECT --reject-with icmp-host-prohibited
create PVEFW-smurflog (2gfT1VMkfr0JL6OccRXTGXo+1qk)
        -A PVEFW-smurflog  -j DROP
create PVEFW-smurfs (HssVe5QCBXd5mc9kC88749+7fag)
        -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
        -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
        -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ip6tables cmdlist:
create GROUP-allow_ceph-IN (7THbUgKtqSW7VYmAamnnyUhaink)
        -A GROUP-allow_ceph-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ceph-OUT (g1sqjMPDECYMWaWoi1LVf1ym+dw)
        -A GROUP-allow_ceph-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_gui-IN (C2ZOZ61Y9OGj0gEkT8Dpm82CpWI)
        -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8)
        -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-IN (4TDlQ1Iq1o6UBZ1EKmiMphvG9fU)
        -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8)
        -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ssh-IN (qUiol7H1++Cjk2TCBJsMuXzJt+8)
        -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM)
        -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-IN (3QdQE7vXIeuhLt/bb+tCKhOVcgY)
        -A GROUP-allow_sync-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-allow_sync-OUT (Uyb+r1bfSlMI9lhniKFzKnxyV9s)
        -A GROUP-allow_sync-OUT -j MARK --set-mark 0x00000000/0x80000000
create PVEFW-Drop (Jb79Uw7z1vZglIcV7QXA5uY/nbk)
        -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Drop  -j PVEFW-DropBroadcast
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
        -A PVEFW-Drop -p udp --dport 137:139 -j DROP
        -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
        -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
        -A PVEFW-Drop -p udp --dport 1900 -j DROP
        -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (8Krk5Nh8pDZOOc7BQAbM6PlyFSU)
        -A PVEFW-DropBroadcast -d ff00::/8 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
        -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
        -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
create PVEFW-HOST-IN (LnkexloL8+lzKK3dH4/gGLuWY0I)
        -A PVEFW-HOST-IN -i lo -j ACCEPT
        -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-advertisement -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-IN -p igmp -j RETURN
        -A PVEFW-HOST-IN -i eno2 -j GROUP-allow_sync-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr2 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr1 -j GROUP-allow_ceph-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_local-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_ssh-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -i vmbr0 -j GROUP-allow_gui-IN
        -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 8006 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 5900:5999 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 3128 -j RETURN
        -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 22 -j RETURN
        -A PVEFW-HOST-IN -j PVEFW-Drop
        -A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (rB/xLO72g/pHiCL0u3sWqn5WwLE)
        -A PVEFW-HOST-OUT -o lo -j ACCEPT
        -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type router-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
        -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
        -A PVEFW-HOST-OUT -p igmp -j RETURN
        -A PVEFW-HOST-OUT -o eno2 -j GROUP-allow_sync-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr2 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr1 -j GROUP-allow_ceph-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_local-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_ssh-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT -o vmbr0 -j GROUP-allow_gui-OUT
        -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
        -A PVEFW-HOST-OUT  -j RETURN
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
        -A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
        -A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (aL1nrxJk/u3XmTb3Am2eaM/3yCM)
        -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
        -A PVEFW-Reject  -j PVEFW-DropBroadcast
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
        -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
        -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
        -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
        -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
        -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
        -A PVEFW-Reject -p udp --dport 1900 -j DROP
        -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
        -A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
        -A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
        -A PVEFW-logflags  -j DROP
create PVEFW-reject (TeZhczhc17LK2pqE7UkGmRMJLNU)
        -A PVEFW-reject -p icmpv6 -j DROP
        -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
        -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
        -A PVEFW-FORWARD -p IPv4 -j ACCEPT
        -A PVEFW-FORWARD -p IPv6 -j ACCEPT
        -A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore FORWARD (zuQi5YOvmMWiM9zohnQw/qWemOA)
ignore INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
detected changes

wolfgang · Dec 7, 2018

Hi,

HE_Cole said:
My Corosync is on a sepreate interface and subnet.

that's fine so you can disable the firewall on this interfaces if they are isolated.
Or allow all traffic from the nodes to the nodes on this interface.

HE_Cole · Dec 7, 2018

wolfgang I never would have thought of that!

But how do i disable it at the interface level ?

Thanks

wolfgang · Dec 10, 2018

You can choose the interface on the rule.
So not choose the interface what corosync useses.

HE_Cole · Dec 12, 2018

Hey wolfgang!

I tried just not adding a rule to the corosync and ceph interfaces but my cluster still drops quroum when i enable the firewall. I only have rules on my vmbr0 which is my WAN interface and did not add rules to other interfaces and yet still enable firewall still brakes ceph and corosync.

With the firewall ON I can still access each node from the pve gui just fine but all nodes are red and have a X and clicking Ceph in the gui under a node just gives a 500 error with the firewall on? And one i turn the firewall on i have to manually turn it off on each node as if i click under data center to turn the firewall off i get error " unable to open file '/etc/pve/firewall/cluster.fw.tmp.2912332' - Permission denied (500)
and i have to go to each node and run pve-firewall stop THEN i can disable to firewall in Data center without error.

Any ideas still?

HE_Cole · Dec 14, 2018

I dont mean to be bugging the communty but i would like some help on this firewall issue i am sure its simple.

I did what wolfgang and i did not any rules for my corosync and ceph interfaces, With the firewall on i can ping on them fine and that part works.

But i still loose quoram and nodes cant ping each other when i turn the firewall on under datacenter. I even have a rule for ALLOW in 0.0.0.0/0 to 0.0.0.0/0 and a ALLOW OUT in postion 0 in the firewall and still my node to node gets blocked.

Here is my pve-firewall

Code:

root@he-s01-r01-pve01:~# pve-firewall compile ipset cmdlist: exists PVEFW-0-management-v4 (LRAv4c0S4qV/TUNkSvLtGKJQpJE) create PVEFW-0-management-v4 hash:net family inet hashsize 64 maxelem 64 add PVEFW-0-management-v4 23.136.0.0/24 exists PVEFW-0-management-v6 (H5WO/Pkuyz4e7OLB2uiMpG0Bsn0) create PVEFW-0-management-v6 hash:net family inet6 hashsize 64 maxelem 64 iptables cmdlist: exists GROUP-allow-any-any-IN (CmRRPmWWGjQB0wNXO+7sGTZqqyI) -A GROUP-allow-any-any-IN -j MARK --set-mark 0x00000000/0x80000000 -A GROUP-allow-any-any-IN -s 0.0.0.0/0 -d 0.0.0.0/0 -g PVEFW-SET-ACCEPT-MARK exists GROUP-allow-any-any-OUT (3SLSpkCfprkcOPMdgrTtP4aXG+g) -A GROUP-allow-any-any-OUT -j MARK --set-mark 0x00000000/0x80000000 -A GROUP-allow-any-any-OUT -s 0.0.0.0/0 -d 0.0.0.0/0 -g PVEFW-SET-ACCEPT-MARK exists GROUP-allow_gui-IN (zNslOg51fEKOPXIk97ncrr1KmUY) -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000 -A GROUP-allow_gui-IN -s 0.0.0.0/0 -d 23.136.0.0/24 -p tcp --sport 1:65535 --dport 8006 -g PVEFW-SET-ACCEPT-MARK exists GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8) -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_local-IN (8KsLa7QApT6AiW4s1PHMXkre9Uc) -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000 -A GROUP-allow_local-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p udp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK -A GROUP-allow_local-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p tcp --sport 1:65535 --dport 1:65535 -g PVEFW-SET-ACCEPT-MARK exists GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8) -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_ssh-IN (abnjYentOoN3Pb779drx14aOswg) -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000 -A GROUP-allow_ssh-IN -s 0.0.0.0/0 -d 23.136.0.0/24 -p tcp --dport 22 -g PVEFW-SET-ACCEPT-MARK exists GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM) -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000 exists PVEFW-Drop (WDy2wbFe7jNYEyoO3QhUELZ4mIQ) -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject -A PVEFW-Drop -j PVEFW-DropBroadcast -A PVEFW-Drop -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT -A PVEFW-Drop -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP -A PVEFW-Drop -p udp --dport 137:139 -j DROP -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP -A PVEFW-Drop -p udp --dport 1900 -j DROP -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Drop -p udp --sport 53 -j DROP exists PVEFW-DropBroadcast (NyjHNAtFbkH7WGLamPpdVnxHy4w) -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP exists PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw) -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT exists PVEFW-FWBR-IN (Ijl7/xz0DD7LF91MlLCz0ybZBE0) -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk) exists PVEFW-HOST-IN (/OOHUEhE0qH9Amh6YKkiDcoF+K8) -A PVEFW-HOST-IN -i lo -j ACCEPT -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs -A PVEFW-HOST-IN -p igmp -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow-any-any-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_local-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_ssh-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_gui-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 8006 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 5900:5999 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 3128 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 22 -j RETURN -A PVEFW-HOST-IN -s 23.136.0.0/24 -d 23.136.0.0/24 -p udp --dport 5404:5405 -j RETURN -A PVEFW-HOST-IN -s 23.136.0.0/24 -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN -A PVEFW-HOST-IN -j PVEFW-Drop -A PVEFW-HOST-IN -j DROP exists PVEFW-HOST-OUT (dyEWVNmhC2MWgSHnilAoV53DDhU) -A PVEFW-HOST-OUT -o lo -j ACCEPT -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-OUT -p igmp -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow-any-any-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_local-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_ssh-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_gui-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 8006 -j RETURN -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 22 -j RETURN -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 5900:5999 -j RETURN -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p tcp --dport 3128 -j RETURN -A PVEFW-HOST-OUT -d 23.136.0.0/24 -p udp --dport 5404:5405 -j RETURN -A PVEFW-HOST-OUT -m addrtype --dst-type MULTICAST -p udp --dport 5404:5405 -j RETURN -A PVEFW-HOST-OUT -j RETURN exists PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk) -A PVEFW-INPUT -j PVEFW-HOST-IN exists PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0) -A PVEFW-OUTPUT -j PVEFW-HOST-OUT exists PVEFW-Reject (CZJnIN6rAdpu+ej59QPr9+laMUo) -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject -A PVEFW-Reject -j PVEFW-DropBroadcast -A PVEFW-Reject -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT -A PVEFW-Reject -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject -A PVEFW-Reject -p udp --dport 1900 -j DROP -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Reject -p udp --sport 53 -j DROP exists PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY) -A PVEFW-SET-ACCEPT-MARK -j MARK --set-mark 0x80000000/0x80000000 exists PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A) -A PVEFW-logflags -j DROP exists PVEFW-reject (Jlkrtle1mDdtxDeI9QaDSL++Npc) -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP -A PVEFW-reject -s 224.0.0.0/4 -j DROP -A PVEFW-reject -p icmp -j DROP -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable -A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited exists PVEFW-smurflog (2gfT1VMkfr0JL6OccRXTGXo+1qk) -A PVEFW-smurflog -j DROP exists PVEFW-smurfs (HssVe5QCBXd5mc9kC88749+7fag) -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog exists PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo) -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags ip6tables cmdlist: exists GROUP-allow-any-any-IN (wOWApKvEMrqx/FoWp4UQoIhRklQ) -A GROUP-allow-any-any-IN -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow-any-any-OUT (wbpUG4+DVKK9sFIDvmWl7xXic28) -A GROUP-allow-any-any-OUT -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_gui-IN (C2ZOZ61Y9OGj0gEkT8Dpm82CpWI) -A GROUP-allow_gui-IN -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_gui-OUT (olTqj37EU0rsvUQExlJCde7vrB8) -A GROUP-allow_gui-OUT -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_local-IN (4TDlQ1Iq1o6UBZ1EKmiMphvG9fU) -A GROUP-allow_local-IN -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_local-OUT (8Yq7dNr1Kg+IUtWmSlM1ItT8JW8) -A GROUP-allow_local-OUT -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_ssh-IN (qUiol7H1++Cjk2TCBJsMuXzJt+8) -A GROUP-allow_ssh-IN -j MARK --set-mark 0x00000000/0x80000000 exists GROUP-allow_ssh-OUT (3xh87Q5ZZEauDwz28xFakYytVPM) -A GROUP-allow_ssh-OUT -j MARK --set-mark 0x00000000/0x80000000 exists PVEFW-Drop (Jb79Uw7z1vZglIcV7QXA5uY/nbk) -A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject -A PVEFW-Drop -j PVEFW-DropBroadcast -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT -A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP -A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP -A PVEFW-Drop -p udp --dport 137:139 -j DROP -A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP -A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP -A PVEFW-Drop -p udp --dport 1900 -j DROP -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Drop -p udp --sport 53 -j DROP exists PVEFW-DropBroadcast (8Krk5Nh8pDZOOc7BQAbM6PlyFSU) -A PVEFW-DropBroadcast -d ff00::/8 -j DROP exists PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw) -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN -A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT exists PVEFW-FWBR-IN (2jmj7l5rSw0yVb/vlWAYkK/YBwk) exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk) exists PVEFW-HOST-IN (1AdmChTabG9C/+Mds0CMQd+F8HA) -A PVEFW-HOST-IN -i lo -j ACCEPT -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-solicitation -j RETURN -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-advertisement -j RETURN -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN -A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN -A PVEFW-HOST-IN -p igmp -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow-any-any-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_local-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_ssh-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -i eno1 -j GROUP-allow_gui-IN -A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 8006 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 5900:5999 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 3128 -j RETURN -A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 22 -j RETURN -A PVEFW-HOST-IN -j PVEFW-Drop -A PVEFW-HOST-IN -j DROP exists PVEFW-HOST-OUT (NsCZQiShTEOT4oQqQSeIIGYwBcU) -A PVEFW-HOST-OUT -o lo -j ACCEPT -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type router-solicitation -j RETURN -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN -A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN -A PVEFW-HOST-OUT -p igmp -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow-any-any-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_local-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_ssh-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -o eno1 -j GROUP-allow_gui-OUT -A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN -A PVEFW-HOST-OUT -j RETURN exists PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk) -A PVEFW-INPUT -j PVEFW-HOST-IN exists PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0) -A PVEFW-OUTPUT -j PVEFW-HOST-OUT exists PVEFW-Reject (aL1nrxJk/u3XmTb3Am2eaM/3yCM) -A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject -A PVEFW-Reject -j PVEFW-DropBroadcast -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT -A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP -A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject -A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject -A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject -A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject -A PVEFW-Reject -p udp --dport 1900 -j DROP -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Reject -p udp --sport 53 -j DROP exists PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY) -A PVEFW-SET-ACCEPT-MARK -j MARK --set-mark 0x80000000/0x80000000 exists PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A) -A PVEFW-logflags -j DROP exists PVEFW-reject (TeZhczhc17LK2pqE7UkGmRMJLNU) -A PVEFW-reject -p icmpv6 -j DROP -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset exists PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo) -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags ebtables cmdlist: exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k) -A PVEFW-FORWARD -p IPv4 -j ACCEPT -A PVEFW-FORWARD -p IPv6 -j ACCEPT -A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT exists PVEFW-FWBR-OUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk) ignore FORWARD (zuQi5YOvmMWiM9zohnQw/qWemOA) ignore INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk) ignore OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk) no changes root@he-s01-r01-pve01:~#

I tryed adding the rule on the interface level and the bridge level and both same issue.

Thanks..

wolfgang · Dec 14, 2018

To debug your Firewall a network config is needed.
send the output of this two commands

Code:

iptables -L
ip a

spirit · Dec 20, 2018

for multicast, source and destination network are not the same

>> -A GROUP-allow_sync-IN -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK
>> -A GROUP-allow_sync-OUT -s 224.0.0.0/4 -d 224.0.0.0/4 -g PVEFW-SET-ACCEPT-MARK

the source is the real ip address of host from were corosync send packets.

(you can verify that with a tcpdump)

HE_Cole · Dec 23, 2018

Issue resolved.

SOLUTION:
I just set the Default INPUT to ALLOW in PVE firewall and then added a BLOCK-ALL 4v/6vTCP/UDP rule at the end of my Allow list and everything works fine on my cluster now and proper traffic is getting blocked and allowed traffic is allowed.

Thanks so much everyone for the help.

Search

Search

[SOLVED] Cluster loses quorum when activating Firewall

HE_Cole

Member

wolfgang

Proxmox Retired Staff

HE_Cole

Member

wolfgang

Proxmox Retired Staff

HE_Cole

Member

HE_Cole

Member

wolfgang

Proxmox Retired Staff

spirit

Distinguished Member

HE_Cole

Member