cluster firewall - change main ip address

Discussion in 'Proxmox VE: Networking and Firewall' started by haukenx, Dec 28, 2018.

  1. haukenx

    haukenx New Member

    Joined:
    Dec 16, 2018
    Messages:
    4
    Likes Received:
    0
    Hi all,

    I installed PVE 5.3-5 on a set of remote hosts, using their public ip-address as main address. Afterwards, I seperated the cluster traffic into another (private) network for pve-traffic (see https://pve.proxmox.com/wiki/Separate_Cluster_Network).

    My cluster firewall still sems to think that the public ip address is still the main one, because it does not block traffic to port 8006 and 22 on the public ip - as it should due to the DROP-policy. I read that there is an auto-rule for that (see https://forum.proxmox.com/threads/problem-getting-pve-firewall-to-work.21333/), but could not find it via iptables -L or in cluster.fw.

    Strangly, the default policy does not seem to be activated. iptables -nvL states: Chain INPUT (policy ACCEPT), though PVE input policy is set to DROP.

    iptables-save:

    Code:
    # Generated by iptables-save v1.6.0 on Fri Dec 28 18:31:04 2018
    *filter
    :INPUT ACCEPT [270526:770969830]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [281449:591056222]
    :PVEFW-Drop - [0:0]
    :PVEFW-DropBroadcast - [0:0]
    :PVEFW-FORWARD - [0:0]
    :PVEFW-FWBR-IN - [0:0]
    :PVEFW-FWBR-OUT - [0:0]
    :PVEFW-INPUT - [0:0]
    :PVEFW-OUTPUT - [0:0]
    :PVEFW-Reject - [0:0]
    :PVEFW-SET-ACCEPT-MARK - [0:0]
    :PVEFW-logflags - [0:0]
    :PVEFW-reject - [0:0]
    :PVEFW-smurflog - [0:0]
    :PVEFW-smurfs - [0:0]
    :PVEFW-tcpflags - [0:0]
    -A INPUT -i vmbr1 -p gre -j ACCEPT
    -A INPUT -j PVEFW-INPUT
    -A FORWARD -j PVEFW-FORWARD
    -A OUTPUT -p gre -j ACCEPT
    -A OUTPUT -j PVEFW-OUTPUT
    -A PVEFW-Drop -p tcp -m tcp --dport 43 -j PVEFW-reject
    -A PVEFW-Drop -j PVEFW-DropBroadcast
    -A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
    -A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
    -A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
    -A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
    -A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
    -A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
    -A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
    -A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
    -A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
    -A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
    -A PVEFW-Drop -m comment --comment "PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ"
    -A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
    -A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
    -A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
    -A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
    -A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
    -A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
    -A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    -A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
    -A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
    -A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
    -A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
    -A PVEFW-FWBR-IN -m comment --comment "PVESIG:Ijl7/xz0DD7LF91MlLCz0ybZBE0"
    -A PVEFW-FWBR-OUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
    -A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
    -A PVEFW-OUTPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"
    -A PVEFW-Reject -p tcp -m tcp --dport 43 -j PVEFW-reject
    -A PVEFW-Reject -j PVEFW-DropBroadcast
    -A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
    -A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
    -A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
    -A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
    -A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
    -A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
    -A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
    -A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
    -A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
    -A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
    -A PVEFW-Reject -m comment --comment "PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo"
    -A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
    -A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
    -A PVEFW-logflags -j DROP
    -A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
    -A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
    -A PVEFW-reject -s 224.0.0.0/4 -j DROP
    -A PVEFW-reject -p icmp -j DROP
    -A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
    -A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
    -A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
    -A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
    -A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
    -A PVEFW-smurflog -j DROP
    -A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
    -A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
    -A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
    -A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
    -A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
    -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
    -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
    -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
    -A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
    -A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
    -A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
    COMMIT
    # Completed on Fri Dec 28 18:31:04 2018
    
    I am confused.

    Kind regards,

    Hauke
     
  2. Richard

    Richard Proxmox Staff Member
    Staff Member

    Joined:
    Mar 6, 2015
    Messages:
    704
    Likes Received:
    25

    It is - the "main one" is that which can be found in "/etc/hosts" leading to the own hostname

    PVE policy DROP is not implemented as iptables -P .. DROP but with a chain having a DROP rule at the end.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. haukenx

    haukenx New Member

    Joined:
    Dec 16, 2018
    Messages:
    4
    Likes Received:
    0
    Hi Richard,

    thanks for your reply.

    I changed /etc/hosts on all nodes to point to the private cluster management addresses, when I created the seperate cluster network according the Wiki. Pinging and ssh-ing both confirm that all hostnames are resolved correctly.

    Thanks for pointing it out that the DROP policy is implemented via a seperate chain. But then, this chain doesn't seem to apply, because as per iptables-save (see original post), I have:

    -A INPUT -i vmbr1 -p gre -j ACCEPT # Added that manually
    -A INPUT -j PVEFW-INPUT

    so, all non-GRE packets are handled in PVEFW-INPUT. But here, we only add a comment.

    -A PVEFW-INPUT -m comment --comment "PVESIG:2jmj7l5rSw0yVb/vlWAYkK/YBwk"

    After that, the default policy (ACCEPT) applies, right? So, the packets never reach any of the PVEFW-Drop*-chains.

    Can you give me a hint to find my mistake?

    Kind regards,

    Hauke
     
  4. Richard

    Richard Proxmox Staff Member
    Staff Member

    Joined:
    Mar 6, 2015
    Messages:
    704
    Likes Received:
    25
    In order to consider the detail it would be useful to know the content of configuration files, as there are:

    /etc/pve/firewall/cluster.fw
    /etc/pve/nodes/*/*.fw

    and compare it with the effect they have seen by
    Code:
    iptables-save
    
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. haukenx

    haukenx New Member

    Joined:
    Dec 16, 2018
    Messages:
    4
    Likes Received:
    0
    Hi Richard,

    all /etc/pve/nodes/*/host.fw look the same:

    Code:
    [OPTIONS]
    
    enable: 0
    
    cluster.fw looks like this (i removed some VM-specific groups, but none of them are referenced in the [RULES]-section, so that should not make a difference).

    vmbr1 is the cluster management bridge. The "outside" world is on vmbr0. If there are already default rules for that, I wouldn't need the pve-mgmt rule at all, right?

    Code:
    [OPTIONS]
    
    policy_in: DROP
    enable: 1
    
    [RULES]
    
    GROUP pve-mgmt -i vmbr1 # Management Traffic
    GROUP nfs # NFS on Storage Transfer Network
    GROUP ceph -i enp179s0f0 # Storage Network
    IN ACCEPT -i vmbr0 -dest x.x.x.x -p udp -dport 1194 # VPN Gateway
    IN ACCEPT -source 192.168.x.0/24 -p tcp -dport 8006 # UI from internal network
    IN SSH(ACCEPT) # SSH
    
    [group ceph] # Ceph storage
    
    OUT ACCEPT -dest 192.168.y.0/24
    IN ACCEPT -source 192.168.y.0/24
    
    [group monitoring] # Monitoring
    
    IN Ping(ACCEPT)
    IN ACCEPT -source x.x.x.x -p tcp -dport 6556 # Input from Monitoring
    
    [group nfs] # Network-File-System
    
    IN ACCEPT -p tcp -dport 2049 # NFS to Server (TCP)
    IN ACCEPT -p udp -dport 2049 # NFS to Server (TCP)
    OUT ACCEPT -p udp -sport 2049 # NFS from Server (UDP)
    OUT ACCEPT -p tcp -sport 2049 # NFS from Server (TCP)
    OUT ACCEPT -p tcp -sport 111 # Portmapper (NFS) from Server (TCP)
    OUT ACCEPT -p udp -sport 111 # Portmapper (NFS) from Server (UDP)
    IN ACCEPT -p udp -dport 111 # Portmapper (NFS) to Server (UDP)
    IN ACCEPT -p tcp -dport 111 # Portmapper (NFS) to Server (TCP)
    
    [group pve-mgmt] # Proxmox Management
    
    OUT ACCEPT -dest 192.168.110.0/24
    IN ACCEPT -source 192.168.110.0/24
    
    
    It seems correct to me. I never touched it manually, only via UI.

    Thanks again for your help!

    Hauke
     
  6. Brat Winget

    Brat Winget New Member

    Joined:
    Jan 3, 2019
    Messages:
    1
    Likes Received:
    0
    The process where it will be going to get the part which will be having the most of it for approving the cluster firewall changes the main IP address that will be guided by the manage all the issue.
     
  7. haukenx

    haukenx New Member

    Joined:
    Dec 16, 2018
    Messages:
    4
    Likes Received:
    0
    Oh dear, oh dear, oh dear.

    I refuse to accept that I have been this stupid :eek:

    I really thought, the host FW does not need to be activated. Shame on me, problem solved. It was a layer-8 problem.

    Kind regards,

    Hauke
     
  8. Rustam Galimov

    Rustam Galimov New Member

    Joined:
    Apr 5, 2019
    Messages:
    1
    Likes Received:
    0
    Thank you! It was useful !
     
  9. EntonyBirk

    EntonyBirk New Member

    Joined:
    Apr 8, 2019
    Messages:
    5
    Likes Received:
    0
    Ooh, this is great! thanks ... it was hard for me to come to this.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice