Settings for Proxmox firewall in cluster mode

[mscd]

Hello together,

I am new to Proxmox VE. At the moment I am testing a cluster setup with two Proxmox nodes in my home LAN and one dedicated server, connected by a IPsec tunnel to my local LAN. In this setup, I would like to use the Proxmox cluster mode primarily to get ONE adminstrative view for all nodes ... the HA features won‘t get used, due to bandwith limitations (40 MBit in the upload direction).
At the moment, I can create a cluster and join the other nodes with success. My problem is, that the cluster gets broken after a while (hours).

To my question(s):
1.) Are there any automatic rules (integrated in the Proxmox firewall), which enable the necessary cluster traffic by default, although the input policy „DROP“ is used?
2.) Which rules should be added to get a working cluster?

Best regards,
mscd

 

[spirit]

Hello together,

I am new to Proxmox VE. At the moment I am testing a cluster setup with two Proxmox nodes in my home LAN and one dedicated server, connected by a IPsec tunnel to my local LAN. In this setup, I would like to use the Proxmox cluster mode primarily to get ONE adminstrative view for all nodes ... the HA features won‘t get used, due to bandwith limitations (40 MBit in the upload direction).
At the moment, I can create a cluster and join the other nodes with success. My problem is, that the cluster gets broken after a while (hours).

To my question(s):
1.) Are there any automatic rules (integrated in the Proxmox firewall), which enable the necessary cluster traffic by default, although the input policy „DROP“ is used?
2.) Which rules should be added to get a working cluster?

Best regards,
mscd

yes, default rules exist for cluster communication (corosync,ssh,..) between nodes

you can check with #iptables-save in HOST-IN rules

(-p udp --dport 5404:5405" , --dport 8006,--dport 5900:5999)

 

[mscd]

Ok ... thanks,

some questions ...

1.) corosync uses UDP-ports (or not?) ... the image below (iptables -nvL) shows that "tcp" is allowed for destination ports 5900-5999
2.) How is "PVFEW-0-management-v4" defined?

Best regards,
mscd

P.S.: To my clarification ... does Proxmox VE 6.3 use unicast or multicast (UDP-ports 5404 & 5405) for corosync to work? I ask this, because I noticed cluster problems with one node connected by a IPsec-VPN-tunnel (in a different subnet), where multicast could not work.

 

[mscd]

... so I have checked it twice ... there seem to be no autogenerated firewall rules for corosync communication to UDP-ports 5404 and 5405 ... could this be the reason, that my cluster crashes recurrently, although the initial join of nodes is working?

Best regards,
mscd

 

[TodorPetkov]

Hello,

can you run "iptables-save" and post results? You can comment out public ip addresses of VMs, if there are any.

 

[mscd]

no problem ... here you are:

# Generated by iptables-save v1.8.2 on Mon Feb 8 19:16:33 2021 *raw :PREROUTING ACCEPT [252498:90939611] :OUTPUT ACCEPT [78032:20785174] COMMIT # Completed on Mon Feb 8 19:16:33 2021 # Generated by iptables-save v1.8.2 on Mon Feb 8 19:16:33 2021 *filter :INPUT ACCEPT [5:320] :FORWARD ACCEPT [19:876] :OUTPUT ACCEPT [21:1276] :PVEFW-Drop - [0:0] :PVEFW-DropBroadcast - [0:0] :PVEFW-FORWARD - [0:0] :PVEFW-FWBR-IN - [0:0] :PVEFW-FWBR-OUT - [0:0] :PVEFW-HOST-IN - [0:0] :PVEFW-HOST-OUT - [0:0] :PVEFW-INPUT - [0:0] :PVEFW-OUTPUT - [0:0] :PVEFW-Reject - [0:0] :PVEFW-SET-ACCEPT-MARK - [0:0] :PVEFW-logflags - [0:0] :PVEFW-reject - [0:0] :PVEFW-smurflog - [0:0] :PVEFW-smurfs - [0:0] :PVEFW-tcpflags - [0:0] -A INPUT -j PVEFW-INPUT -A FORWARD -j PVEFW-FORWARD -A OUTPUT -j PVEFW-OUTPUT -A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject -A PVEFW-Drop -j PVEFW-DropBroadcast -A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT -A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP -A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP -A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP -A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP -A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP -A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Drop -p udp -m udp --sport 53 -j DROP -A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ" -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP -A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w" -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN -A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT -A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw" -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs -A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0" -A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk" -A PVEFW-HOST-IN -i lo -j ACCEPT -A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs -A PVEFW-HOST-IN -p igmp -j RETURN -A PVEFW-HOST-IN -p tcp -m tcp --dport 22 -j RETURN -A PVEFW-HOST-IN -p tcp -m tcp --dport 8006 -j RETURN -A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN -A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN -A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN -A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN -A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN -A PVEFW-HOST-IN -j PVEFW-Drop -A PVEFW-HOST-IN -j DROP -A PVEFW-HOST-IN -m comment --comment "PVESIG:6Ss/RZEdBj97jVOcA5SJrWhg/RA" -A PVEFW-HOST-OUT -o lo -j ACCEPT -A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP -A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A PVEFW-HOST-OUT -p igmp -j RETURN -A PVEFW-HOST-OUT -d 172.16.1.0/24 -p tcp -m tcp --dport 8006 -j RETURN -A PVEFW-HOST-OUT -d 172.16.1.0/24 -p tcp -m tcp --dport 22 -j RETURN -A PVEFW-HOST-OUT -d 172.16.1.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN -A PVEFW-HOST-OUT -d 172.16.1.0/24 -p tcp -m tcp --dport 3128 -j RETURN -A PVEFW-HOST-OUT -j RETURN -A PVEFW-HOST-OUT -m comment --comment "PVESIG:dDsZir5ROEJK8EXBKRRpHQCahOo" -A PVEFW-INPUT -j PVEFW-HOST-IN -A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk" -A PVEFW-OUTPUT -j PVEFW-HOST-OUT -A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0" -A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject -A PVEFW-Reject -j PVEFW-DropBroadcast -A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT -A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP -A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject -A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject -A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject -A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject -A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP -A PVEFW-Reject -p udp -m udp --sport 53 -j DROP -A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo" -A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000 -A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY" -A PVEFW-logflags -j DROP -A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A" -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP -A PVEFW-reject -s 224.0.0.0/4 -j DROP -A PVEFW-reject -p icmp -j DROP -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable -A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited -A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc" -A PVEFW-smurflog -j DROP -A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk" -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog -A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag" -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags -A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo" COMMIT # Completed on Mon Feb 8 19:16:33 2021

 

[mscd]

Update ... the necessary rules (UDP 5404 and UDP 5405) are getting active AFTER creating a corresponding cluster (and the join of some nodes) ... so the rules do not exist in the iptables-view of a local node (after a fresh installation).