[SOLVED] local_network alias for multiple interfaces with redundant ring protocol

EuroDomenii

Well-Known Member
Sep 30, 2016
144
30
48
Slatina
www.domenii.eu
According to https://pve.proxmox.com/wiki/Firewall#pve_firewall_ip_aliases, user can overwrite local_network in the cluster.fw , in order for the firewall automatically sets up rules to allow everything needed for cluster communication (corosync, API, SSH) using this alias.

What about https://pve.proxmox.com/wiki/Separate_Cluster_Network#Redundant_Ring_Protocol ?

Use case: public ip addresses from different subnets, mixed with private subnet.

Code:
cat /etc/corosync/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: rrp3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: rrp3
    ring1_addr: rrp3private
  }

  node {
    name: rrp2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: rrp2
    ring1_addr: rrp2private
  }

  node {
    name: rrp1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: rrp1
    ring1_addr: rrp1private
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: eurodomenii
  config_version: 11
  rrp_mode: passive
  transport: udpu
  ip_version: ipv4
  secauth: on
  version: 2
  interface {
    ringnumber: 0
  }
  interface {
    bindnetaddr: 192.168.0.0
    ringnumber: 1
  }

}

For public ip addresses I have to manually setup the firewall rules, or is there an elegant setup, like multiple local_network values?
 
Last edited:
SHORT STORY

Solution for use case with public ip address, from different subnets ( /etc/pve/firewall/cluster.fw)

Code:
[RULES]

IN ACCEPT -source +management -dest +management -p udp -dport 5404:5405
OUT ACCEPT -source +management -dest +management -p udp -sport 5404:5405

LONG STORY

Cluster description: rrp1, rrp2, rrp3 public ip addresses, and rrp1private, rrp2private, rrp3private are corresponding private ip addresses from [ALIASES] local_network 192.168.0.0/24

Testing use case: ifdown rrp3private, I still need to have a valid 3 node cluster, with firewall enabled.

a) If rrp1, rrp2, rrp3 are all from the same subnet, it works out of the box

Code:
pve-firewall localnet
local hostname: rrp1
local IP address: rrp1_ip_address
network auto detect: rrp1_subnet
using user defined local_network: 192.168.0.0/24
b) If rrp1, rrp2, rrp3 are public ips from different subnets, it doesn’t work out of the box

Code:
pve-firewall localnet
local hostname: rrp1
local IP address: rrp1_ip_address
network auto detect: 127.0.0.0/8
using user defined local_network: 192.168.0.0/24

The local cluster network is automatically added to this IP set (alias cluster_network), to enable inter-host cluster communication. (multicast,ssh,…) via https://pve.proxmox.com/wiki/Firewall#pve_firewall_ip_sets. But, if we take a look to default firewall rules set, ip set management has rules for necessary ports, except 5404:5405 for corosync.

Code:
Chain PVEFW-HOST-IN (1 references)

  311 16172 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:8006
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpts:5900:5999
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:3128
    3   208 RETURN     tcp  --  any    any     anywhere             anywhere             match-set PVEFW-0-management-v4 src tcp dpt:ssh
    0     0 RETURN     udp  --  any    any     192.168.0.0/24       192.168.0.0/24       udp dpts:5404:5405
    0     0 RETURN     udp  --  any    any     192.168.0.0/24       anywhere             ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405

My workaround is to add custom rules, for management ip set, for corosync ports, but I guess a patch should be required to the default firewall rules set, for this use case.

I don’t need multicast rules, because public network works on unicast.

Code:
cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 1

[ALIASES]

local_network 192.168.0.0/24

[IPSET management]

rrp1 ip
rrp2 ip
rrp3 ip
etc

[RULES]

OUT ACCEPT -dest +management -p udp -dport 5404:5405
IN ACCEPT -source +management -dest +management -p udp -dport 5404:5405

The output of the new rules ( in bold letters, without code tag) .

root@rrp1:~# iptables -vL | grep 5404
824K 371M RETURN udp -- any any anywhere anywhere match-set PVEFW-0-management-v4 src match-set PVEFW-0-management-v4 dst udp dpts:5404:5405
0 0 RETURN udp -- any any 192.168.0.0/24 192.168.0.0/24 udp dpts:5404:5405
0 0 RETURN udp -- any any 192.168.0.0/24 anywhere ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405
3057K 696M RETURN udp -- any any anywhere anywhere match-set PVEFW-0-management-v4 dst udp dpts:5404:5405
0 0 RETURN udp -- any any anywhere 192.168.0.0/24 udp dpts:5404:5405
0 0 RETURN udp -- any any anywhere anywhere ADDRTYPE match dst-type MULTICAST udp dpts:5404:5405


root@rrp1:~# iptables-save | grep 5404
-A PVEFW-HOST-IN -p udp -m set --match-set PVEFW-0-management-v4 src -m set --match-set PVEFW-0-management-v4 dst -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.0.0/24 -d 192.168.0.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.0.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m set --match-set PVEFW-0-management-v4 dst -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.0.0/24 -p udp -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!