cluster firewall - change main ip address

H

haukenx

Active Member
Dec 16, 2018
23
1
43
47
Hi all,

I installed PVE 5.3-5 on a set of remote hosts, using their public ip-address as main address. Afterwards, I seperated the cluster traffic into another (private) network for pve-traffic (see https://pve.proxmox.com/wiki/Separate_Cluster_Network).

My cluster firewall still sems to think that the public ip address is still the main one, because it does not block traffic to port 8006 and 22 on the public ip - as it should due to the DROP-policy. I read that there is an auto-rule for that (see https://forum.proxmox.com/threads/problem-getting-pve-firewall-to-work.21333/), but could not find it via iptables -L or in cluster.fw.

Strangly, the default policy does not seem to be activated. iptables -nvL states: Chain INPUT (policy ACCEPT), though PVE input policy is set to DROP.

iptables-save:

Code: 
# Generated by iptables-save v1.6.0 on Fri Dec 28 18:31:04 2018
*filter
:INPUT ACCEPT [270526:770969830]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [281449:591056222]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -i vmbr1 -p gre -j ACCEPT
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -p gre -j ACCEPT
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-OUTPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
-A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Fri Dec 28 18:31:04 2018

I am confused.

Kind regards,

Hauke
 
haukenx said:
My cluster firewall still sems to think that the public ip address is still the main one
Click to expand...


It is - the "main one" is that which can be found in "/etc/hosts" leading to the own hostname

haukenx said:
Strangly, the default policy does not seem to be activated. iptables -nvL states: Chain INPUT (policy ACCEPT), though PVE input policy is set to DROP.
Click to expand...

PVE policy DROP is not implemented as iptables -P .. DROP but with a chain having a DROP rule at the end.
 
Hi Richard,

thanks for your reply.

I changed /etc/hosts on all nodes to point to the private cluster management addresses, when I created the seperate cluster network according the Wiki. Pinging and ssh-ing both confirm that all hostnames are resolved correctly.

Thanks for pointing it out that the DROP policy is implemented via a seperate chain. But then, this chain doesn't seem to apply, because as per iptables-save (see original post), I have:

-A INPUT -i vmbr1 -p gre -j ACCEPT # Added that manually
-A INPUT -j PVEFW-INPUT

so, all non-GRE packets are handled in PVEFW-INPUT. But here, we only add a comment.

-A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"

After that, the default policy (ACCEPT) applies, right? So, the packets never reach any of the PVEFW-Drop*-chains.

Can you give me a hint to find my mistake?

Kind regards,

Hauke
 
haukenx said:
After that, the default policy (ACCEPT) applies, right? So, the packets never reach any of the PVEFW-Drop*-chains.

Can you give me a hint to find my mistake?
Click to expand...

In order to consider the detail it would be useful to know the content of configuration files, as there are:

/etc/pve/firewall/cluster.fw
/etc/pve/nodes/*/*.fw

and compare it with the effect they have seen by
Code: 
iptables-save
 
Hi Richard,

all /etc/pve/nodes/*/host.fw look the same:

Code: 
[OPTIONS]

enable: 0

cluster.fw looks like this (i removed some VM-specific groups, but none of them are referenced in the [RULES]-section, so that should not make a difference).

vmbr1 is the cluster management bridge. The "outside" world is on vmbr0. If there are already default rules for that, I wouldn't need the pve-mgmt rule at all, right?

Code: 
[OPTIONS]

policy_in: DROP
enable: 1

[RULES]

GROUP pve-mgmt -i vmbr1 # Management Traffic
GROUP nfs # NFS on Storage Transfer Network
GROUP ceph -i enp179s0f0 # Storage Network
IN ACCEPT -i vmbr0 -dest x.x.x.x -p udp -dport 1194 # VPN Gateway
IN ACCEPT -source 192.168.x.0/24 -p tcp -dport 8006 # UI from internal network
IN SSH(ACCEPT) # SSH

[group ceph] # Ceph storage

OUT ACCEPT -dest 192.168.y.0/24
IN ACCEPT -source 192.168.y.0/24

[group monitoring] # Monitoring

IN Ping(ACCEPT)
IN ACCEPT -source x.x.x.x -p tcp -dport 6556 # Input from Monitoring

[group nfs] # Network-File-System

IN ACCEPT -p tcp -dport 2049 # NFS to Server (TCP)
IN ACCEPT -p udp -dport 2049 # NFS to Server (TCP)
OUT ACCEPT -p udp -sport 2049 # NFS from Server (UDP)
OUT ACCEPT -p tcp -sport 2049 # NFS from Server (TCP)
OUT ACCEPT -p tcp -sport 111 # Portmapper (NFS) from Server (TCP)
OUT ACCEPT -p udp -sport 111 # Portmapper (NFS) from Server (UDP)
IN ACCEPT -p udp -dport 111 # Portmapper (NFS) to Server (UDP)
IN ACCEPT -p tcp -dport 111 # Portmapper (NFS) to Server (TCP)

[group pve-mgmt] # Proxmox Management

OUT ACCEPT -dest 192.168.110.0/24
IN ACCEPT -source 192.168.110.0/24

It seems correct to me. I never touched it manually, only via UI.

Thanks again for your help!

Hauke
 
The process where it will be going to get the part which will be having the most of it for approving the cluster firewall changes the main IP address that will be guided by the manage all the issue.
 
Oh dear, oh dear, oh dear.

I refuse to accept that I have been this stupid :eek:

I really thought, the host FW does not need to be activated. Shame on me, problem solved. It was a layer-8 problem.

Kind regards,

Hauke
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom