Firewall configuration issues after adding additional cluster link

R

ret

New Member
Proxmox Subscriber
Mar 29, 2026
3
0
1
Hello,

I have a question regarding the firewall configuration when adding an additional link for replication/clustering between Proxmox nodes.

I followed the official documentation:
The additional link was successfully added, but the firewall configuration does not seem to be correct.

On both nodes, I can see that a new iptables rule for corosync traffic has been created automatically. However, it appears to use the wrong port (5406 instead of 5404–5405). In addition, there are no firewall rules for SSH on the new link, which seems to be the reason why migration over this link is not working.

Did I miss something in the configuration, or is this a known issue?

I am aware that I can add the required firewall rules manually. However, since rules are being added automatically (albeit incorrectly), I would expect this to work without manual intervention.

Any guidance would be appreciated.

Thank you.

Bash: 
root@server-01:/home/tb# journalctl -b -u corosync
Mar 23 23:15:20 server-01 corosync[687304]:   [CFG   ] Config reload requested by node 1
Mar 23 23:15:20 server-01 corosync[687304]:   [TOTEM ] Configuring link 0
Mar 23 23:15:20 server-01 corosync[687304]:   [TOTEM ] Configured link number 0: local addr: 10.18.7.101, port=5405
Mar 23 23:15:20 server-01 corosync[687304]:   [QUORUM] This node is within the non-primary component and will NOT provide any services.
Mar 23 23:15:20 server-01 corosync[687304]:   [QUORUM] Members[1]: 1
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 0)
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 has no active links
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 has no active links
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] host: host: 2 has no active links
Mar 23 23:15:20 server-01 corosync[687304]:   [KNET  ] pmtud: MTU manually set to: 0
Mar 23 23:15:23 server-01 corosync[687304]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Mar 23 23:15:23 server-01 corosync[687304]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Mar 23 23:15:23 server-01 corosync[687304]:   [QUORUM] Sync members[2]: 1 2

root@server-02:/home/tb# journalctl -b -u corosync
Mar 29 18:39:02 server-02 corosync[2809]:   [CFG   ] Config reload requested by node 1
Mar 29 18:39:02 server-02 corosync[2809]:   [TOTEM ] Configuring link 0
Mar 29 18:39:02 server-02 corosync[2809]:   [TOTEM ] Configured link number 0: local addr: 10.18.7.102, port=5405
Mar 29 18:39:02 server-02 corosync[2809]:   [TOTEM ] Configuring link 1
Mar 29 18:39:02 server-02 corosync[2809]:   [TOTEM ] Configured link number 1: local addr: 10.18.16.102, port=5406
Mar 29 18:39:02 server-02 corosync[2809]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Mar 29 18:39:02 server-02 corosync[2809]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Mar 29 18:39:02 server-02 corosync[2809]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Mar 29 18:39:02 server-02 corosync[2809]:   [KNET  ] pmtud: MTU manually set to: 0
Mar 29 18:39:04 server-02 corosync[2809]:   [KNET  ] rx: host: 1 link: 1 is up
Mar 29 18:39:04 server-02 corosync[2809]:   [KNET  ] link: Resetting MTU for link 1 because host 1 joined
Mar 29 18:39:04 server-02 corosync[2809]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Mar 29 18:39:04 server-02 corosync[2809]:   [KNET  ] pmtud: PMTUD link change for host: 1 link: 1 from 469 to 8885
Mar 29 18:39:04 server-02 corosync[2809]:   [KNET  ] pmtud: Global data MTU changed to: 1397

root@server-01:/home/tb# corosync-cfgtool -sb
Local node ID 1, transport knet
LINK ID 0 udp
    addr    = 10.18.7.101
    status    = n3
LINK ID 1 udp
    addr    = 10.18.16.101
    status    = n3

root@server-02:/home/tb# corosync-cfgtool -sb
Local node ID 2, transport knet
LINK ID 0 udp
    addr    = 10.18.7.102
    status    = 3n
LINK ID 1 udp
    addr    = 10.18.16.102
    status    = 3n

Bash: 
root@server-01:/home/tb# ss -tulpen | grep :54
udp   UNCONN 0      0        10.18.7.101:5405      0.0.0.0:*    users:(("corosync",pid=3204788,fd=28))
udp   UNCONN 0      0       10.18.16.101:5406      0.0.0.0:*    users:(("corosync",pid=3204788,fd=34))

root@server-01:/home/tb# iptables -nvL | grep :54
    0     0 RETURN     udp  --  *      *       10.18.7.102          10.18.7.101          udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.16.102         10.18.16.101         udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.7.101          10.18.7.102          udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.16.101         10.18.16.102         udp dpts:5404:5405
root@server-01:/home/tb# iptables -nvL | grep :22
    0     0 PVEFW-SET-ACCEPT-MARK  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           [goto]  match-set PVEFW-0-checkmk-v4 src tcp dpt:22
    1    60 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set PVEFW-0-management-v4 src tcp dpt:22
    1    60 RETURN     tcp  --  *      *       0.0.0.0/0            10.18.7.0/24         tcp dpt:22

root@server-01:/home/tb# ipset list PVEFW-0-management-v4
Name: PVEFW-0-management-v4
Type: hash:net
Revision: 7
Header: family inet hashsize 64 maxelem 64 bucketsize 12 initval 0x2bf959ba
Size in memory: 504
References: 5
Number of entries: 1
Members:
10.18.7.0/24


root@server-02:/home/tb# ss -tulpen | grep :54
udp   UNCONN 0      0        10.18.7.102:5405      0.0.0.0:*    users:(("corosync",pid=2809,fd=28))
udp   UNCONN 0      0       10.18.16.102:5406      0.0.0.0:*    users:(("corosync",pid=2809,fd=33))

root@server-02:/home/tb# iptables -nvL | grep :54
    0     0 RETURN     udp  --  *      *       10.18.7.101          10.18.7.102          udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.16.101         10.18.16.102         udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.7.102          10.18.7.101          udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.16.102         10.18.16.101         udp dpts:5404:5405
root@server-02:/home/tb# iptables -nvL | grep :22
    0     0 PVEFW-SET-ACCEPT-MARK  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           [goto]  match-set PVEFW-0-checkmk-v4 src tcp dpt:22
    1    60 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set PVEFW-0-management-v4 src tcp dpt:22
    1    60 RETURN     tcp  --  *      *       0.0.0.0/0            10.18.7.0/24         tcp dpt:22

root@server-02:/home/tb# ipset list PVEFW-0-management-v4
Name: PVEFW-0-management-v4
Type: hash:net
Revision: 7
Header: family inet hashsize 64 maxelem 64 bucketsize 12 initval 0xe43b1bda
Size in memory: 504
References: 5
Number of entries: 1
Members:
10.18.7.0/24
 
Hi SteveITS,

Thank you for your reply. Of course, corosync for coordinating which cluster node is online and migration between the nodes via SSH are two different functions and protocols. However, both are required for proper cluster operation.

From my side, there are two separate questions:
  1. Regarding corosync: Why is a firewall rule for corosync on the second link added automatically, which I appreciate, but with the wrong destination port range of 5404:5405 instead of 5406?
  2. Regarding SSH: Since the iptables rules and ipset lists that allow migration between hosts over the initially used link are created automatically using the ipset list PVEFW-0-management-v4 (even if its primary purpose is not node migration), I assumed that rules for the new link would also be added automatically.
That is why I am asking whether I am doing something wrong, or whether firewall rules to allow SSH traffic between cluster nodes must be added manually.

BR
Thomas
 
I see what you are saying. Yet corosync-cfgtool -n shows all links are connected, here.

Since each link is a unique IP there should be no reason to change ports...?
 
Right. But the rules created use "udp dpts:5404:5405" which actually includes 5404, and omit other ports. Old post https://forum.proxmox.com/threads/firewall-ports-cluster-configuration.16210/post-83391 mentions 5404 along with the linked https://pve.proxmox.com/wiki/Ports (PVE <6) but not https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_ports_used_by_proxmox_ve.

It turns out we have a datacenter fw rule allowing the corosync IP ranges which should cover it. I don't know if that was added to make it work? And/or to allow other cross-node access. I may be blind but I'm not seeing that in iptables' rules either.

@ret What is your corosync-cfgtool -n output?
 
I assume the 5405-5405 ports are leftovers from Corosync 2. Corosync 3 is covered by the default OUT policy accept and for receiving conntrack jumps in by assuming the connection is established and lets the traffic through. I have no idea, if this is the intended way. Looks awkward though.
 
Hi,

Thank you all for the valuable input.

@SteveITS
As we can see, both links are indeed connected. This was not clear to me from the "corosync-cfgtool -sb" output.

Code: 
root@server-01:/home/tb# corosync-cfgtool -n
Local node ID 1, transport knet
nodeid: 2 reachable
   LINK: 0 udp (10.18.7.101->10.18.7.102) enabled connected mtu: 1397
   LINK: 1 udp (10.18.16.101->10.18.16.102) enabled connected mtu: 8885

root@server-02:/home/tb# corosync-cfgtool -n
Local node ID 2, transport knet
nodeid: 1 reachable
   LINK: 0 udp (10.18.7.102->10.18.7.101) enabled connected mtu: 1397
   LINK: 1 udp (10.18.16.102->10.18.16.101) enabled connected mtu: 8885

@fba
I think you are also right about the 5404:5405 iptables rules. After several days of uptime, these rules still show 0 matches. This indicates that Corosync traffic is being accepted earlier in the chain.

Code: 
    0     0 RETURN     udp  --  *      *       10.18.7.101          10.18.7.102          udp dpts:5404:5405
    0     0 RETURN     udp  --  *      *       10.18.16.101         10.18.16.102         udp dpts:5404:5405


So, in the end, my understanding is this:
  • Individual iptables rules for Corosync are still being created, but they appear to be leftovers from an older Corosync version and therefore are no longer used.
  • Traffic for the current Corosync version is accepted earlier in the iptables chain. These rules appear to cover all traffic for any Corosync link.
  • iptables rules for SSH are created for the first management IP on every Proxmox node. Their main purpose is to allow management access to the cluster nodes. These rules are also defined in such a way that VM migration works via the first configured cluster link.
  • Rules for SSH connections between cluster nodes on additional cluster links need to be created manually, if migration between those links is required.
  • Additionally, destination ports 60000:60050 (TCP) must be reachable between nodes for live migration.
The easiest approach to add all required iptables rules for enabling migration via the additional link is to create an IP set called "management" (name it exactly like this) in the Datacenter context. All mentioned networks will then be automatically added to the PVEFW-0-management-v4 ipset list. This allows the required ports: 8006, 5900:5999, 3128, 22, and 60000:60050. Please note that ports 8006, 5900:5999 and 3128 are not needed for cluster operations.

BR
Thomas
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back