[SOLVED] local_network alias for multiple interfaces with redundant ring protocol

[EuroDomenii]

According to https://pve.proxmox.com/wiki/Firewall#pve_firewall_ip_aliases, user can overwrite local_network in the cluster.fw , in order for the firewall automatically sets up rules to allow everything needed for cluster communication (corosync, API, SSH) using this alias.

What about https://pve.proxmox.com/wiki/Separate_Cluster_Network#Redundant_Ring_Protocol ?

Use case: public ip addresses from different subnets, mixed with private subnet.

cat /etc/corosync/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: rrp3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: rrp3
    ring1_addr: rrp3private
  }

  node {
    name: rrp2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: rrp2
    ring1_addr: rrp2private
  }

  node {
    name: rrp1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: rrp1
    ring1_addr: rrp1private
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: eurodomenii
  config_version: 11
  rrp_mode: passive
  transport: udpu
  ip_version: ipv4
  secauth: on
  version: 2
  interface {
    ringnumber: 0
  }
  interface {
    bindnetaddr: 192.168.0.0
    ringnumber: 1
  }

}

For public ip addresses I have to manually setup the firewall rules, or is there an elegant setup, like multiple local_network values?

 

[EuroDomenii]

SHORT STORY

Solution for use case with public ip address, from different subnets ( /etc/pve/firewall/cluster.fw)

[RULES]

IN ACCEPT -source +management -dest +management -p udp -dport 5404:5405
OUT ACCEPT -source +management -dest +management -p udp -sport 5404:5405

LONG STORY

Cluster description: rrp1, rrp2, rrp3 public ip addresses, and rrp1private, rrp2private, rrp3private are corresponding private ip addresses from [ALIASES] local_network 192.168.0.0/24

Testing use case: ifdown rrp3private, I still need to have a valid 3 node cluster, with firewall enabled.

a) If rrp1, rrp2, rrp3 are all from the same subnet, it works out of the box

pve-firewall localnet
local hostname: rrp1
local IP address: rrp1_ip_address
network auto detect: rrp1_subnet
using user defined local_network: 192.168.0.0/24

b) If rrp1, rrp2, rrp3 are public ips from different subnets, it doesn’t work out of the box

pve-firewall localnet
local hostname: rrp1
local IP address: rrp1_ip_address
network auto detect: 127.0.0.0/8
using user defined local_network: 192.168.0.0/24

The local cluster network is automatically added to this IP set (alias cluster_network), to enable inter-host cluster communication. (multicast,ssh,…) via https://pve.proxmox.com/wiki/Firewall#pve_firewall_ip_sets. But, if we take a look to default firewall rules set, ip set management has rules for necessary ports, except 5404:5405 for corosync.

Chain PVEFW-HOST-IN (1 references)

  311 16172 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:8006
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpts:5900:5999
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:3128
    3   208 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:ssh
    0     0 RETURN     udp  --  any    any     192.168.0.0/24       192.168.0.0/24       udp dpts:5404:5405
    0     0 RETURN     udp  --  any    any     192.168.0.0/24       anywhere             ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405

My workaround is to add custom rules, for management ip set, for corosync ports, but I guess a patch should be required to the default firewall rules set, for this use case.

I don’t need multicast rules, because public network works on unicast.

cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 1

[ALIASES]

local_network 192.168.0.0/24

[IPSET management]

rrp1 ip
rrp2 ip
rrp3 ip
etc

[RULES]

OUT ACCEPT -dest +management -p udp -dport 5404:5405
IN ACCEPT -source +management -dest +management -p udp -dport 5404:5405

The output of the new rules ( in bold letters, without code tag) .

root@rrp1:~# iptables -vL | grep 5404
824K 371M RETURN udp -- any any anywhere anywhere match-set PVEFW-0-management-v4 src match-set PVEFW-0-management-v4 dst udp dpts:5404:5405
0 0 RETURN udp -- any any 192.168.0.0/24 192.168.0.0/24 udp dpts:5404:5405
0 0 RETURN udp -- any any 192.168.0.0/24 anywhere ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405
3057K 696M RETURN udp -- any any anywhere anywhere match-set PVEFW-0-management-v4 dst udp dpts:5404:5405
0 0 RETURN udp -- any any anywhere 192.168.0.0/24 udp dpts:5404:5405
0 0 RETURN udp -- any any anywhere anywhere ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405


root@rrp1:~# iptables-save | grep 5404
-A PVEFW-HOST-IN -p udp -m set --match-set PVEFW-0-management-v4 src -m set --match-set PVEFW-0-management-v4 dst -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.0.0/24 -d 192.168.0.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.0.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m set --match-set PVEFW-0-management-v4 dst -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.0.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN