Host based firewall - corosync issues (udpu)

K

kristian.kirilov

Well-Known Member
Nov 17, 2016
64
2
48
39
Hello,

I'm trying to set up host based firewall. There are two hosts which run Proxmox (latest version) and I just want to secure them via host based firewall. I understand that the firewall is applied in that way - datacenter -> hosts -> vms. So I made some ip sets for easy recognition of the host entries and easy reconfiguration in case of ip change. The issues I'm hitting is based on corosync cluster communication (1) and some strange errors in logs for undefined variables (2).

I will post some configs to ensure everybody understand what I'm saying.

Code: 
root@proxmox-node-1.home.lan:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: proxmox-node-1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: proxmox-node-1
    ring1_addr: proxmox-node-1c
  }
  node {
    name: proxmox-node-2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: proxmox-node-2
    ring1_addr: proxmox-node-2c
  }
}

quorum {
  provider: corosync_votequorum
  two_node: 1
}

totem {
  cluster_name: proxmox
  transport: udpu
  rrp_mode: passive
  config_version: 5
  interface {
    bindnetaddr: 192.168.0.0
    ringnumber: 0
  }
  interface {
    bindnetaddr: 10.9.8.0
    ringnumber: 1
  }
  ip_version: ipv4
  secauth: on
  version: 2
}
root@proxmox-node-1.home.lan:~#

So there are two network card, eth0 is the external network card, which is connected to the switch. It works as a trunk port and delivers traffic to VM's.

eth1 is the internal card which is directly connected to the other server via back to back technic.
So the ip addresses of the network cards is as follow: eth0 - 192.168.0.2 and eth1 is 10.9.8.6.
There is also two ring buffers for these separated networks.

Here is the configuration of the firewall:

Code: 
root@proxmox-node-1.home.lan:~# cat /etc/pve/firewall/cluster.fw
[OPTIONS]

enable: 0
ebtables: 0

[ALIASES]

nagios_server 192.168.0.9 # The Nagios3 server
dns_server 192.168.0.7 # The DNS server
monitoring_server 192.168.0.10 # The Monitoring server
puppet_server 192.168.10.18 # The Puppet server
qnap_storage 192.168.0.5 # The QNAP storage
firebat_server 192.168.10.9 # The Firebat server

[IPSET dmz_network] # Vlan3 (192.168.10.0/24)

192.168.10.0/24 # DMZ Network

[IPSET guest_network] # Vlan4 (192.168.20.0/24)

192.168.20.0/24 # Guest Network

[IPSET lan_network] # Vlan2 (192.168.0.0/24)

192.168.0.0/24 # Lan Network

[IPSET management_ipset] # Trusted IP's of KpuCko's computers

192.168.0.15 # KpuCko's Win10 Tower PC
192.168.0.16 # KpuCko's T430S Laptop PC

[IPSET management_network] # Vlan5 (192.168.30.0/24)

192.168.30.0/24 # Management Network

[IPSET nodes_ipset_external] # Access to Proxmox servers

192.168.0.2 # Proxmox Node 1 (external interface)
192.168.0.23 # Proxmox VIP (VRRP interface)
192.168.0.3 # Proxmox Node 2 (external interface)

[IPSET nodes_ipset_internal] # Access to the Proxmox servers

10.9.8.6 # Proxmox Node 1 (internal interface)
10.9.8.7 # Proxmox Node 2 (internal interface)

[RULES]

IN ACCEPT -p icmp # Allow ICMP protocol
IN DNS(ACCEPT) -dest dns_server # Allow DNS to the DNS server
IN ACCEPT -source monitoring_server -p udp -dport 8089 # Allow InfluxDB to the Monitoring server
IN SNMP(ACCEPT) -source monitoring_server # Allow SNMP to all nodes
IN SMTP(ACCEPT) -dest firebat_server # Allow SMTP to the Firebat server
IN ACCEPT -source nagios_server -p tcp -dport 5666 # Allow NRPE requests from the Nagios3 server
IN Munin(ACCEPT) -source nagios_server # Allow Munin requests from Nagios3 server

[group management] # Rule to allow management access to the Proxmox servers

IN ACCEPT -source +nodes_ipset_internal -dest +nodes_ipset_internal # Allow internal communication (corosync, ssh)
IN ACCEPT -source +nodes_ipset_external -dest +nodes_ipset_external # Allow external communication (corosync, ssh)
IN ACCEPT -source +management_ipset -dest +nodes_ipset_external -p tcp -dport 2224 # Allow Pacemaker WebGUI to the nodes
IN ACCEPT -source +management_ipset -dest +nodes_ipset_external -p tcp -dport 3128 # Allow VNC to the nodes
IN Web(ACCEPT) -source +management_ipset -dest +nodes_ipset_external # Allow WEB to the nodes
IN ACCEPT -source +management_ipset -dest +nodes_ipset_external -p tcp -dport 8006 # Allow WebGUI access to the nodes
IN SSH(ACCEPT) -source +management_ipset -dest +nodes_ipset_external # Allow SSH to the nodes
IN ACCEPT -source +management_ipset -dest +nodes_ipset_external -p tcp -dport 5900:5999 # Allow VNC Web console

root@proxmox-node-1.home.lan:~#

The host firewall:

Code: 
root@proxmox-node-1.home.lan:~# cat /etc/pve/nodes/proxmox-node-1/host.fw
[RULES]

IN ACCEPT -dest +nodes_ipset_external -p udp -dport 5404:5405 # Allow Corosync to external network card
IN ACCEPT -source +nodes_ipset_external -p udp -dport 5404:5405 # Allow Corosync to external network card
GROUP management # Allow Management access to the nodes
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 32764:32769 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 32764:32769 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 2049 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 2049 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 111 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 111 # Allow NFS to the QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 3493 # Allow NUT to QNAP storage
IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 3493 # Allow NUT to QNAP storage
IN ACCEPT -source qnap_storage -dest +nodes_ipset_external -p udp -dport 3493 # Allow NUT to QNAP storage
IN ACCEPT -source qnap_storage -dest +nodes_ipset_external -p tcp -dport 3493 # Allow NUT to QNAP storage

root@proxmox-node-1.home.lan:~#

So when I start the firewall and reboot the boxes, I'm watching the syslog and get these errors:

Code: 
Jan 12 09:40:38 proxmox-node-1 corosync[2561]: error   [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:40:38 proxmox-node-1 corosync[2561]:  [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:40:39 proxmox-node-1 corosync[2561]: notice  [TOTEM ] Automatically recovered ring 0
Jan 12 09:40:39 proxmox-node-1 corosync[2561]:  [TOTEM ] Automatically recovered ring 0
Jan 12 09:41:00 proxmox-node-1 systemd[1]: Starting Proxmox VE replication runner...
Jan 12 09:41:00 proxmox-node-1 systemd[1]: Started Proxmox VE replication runner.
Jan 12 09:41:05 proxmox-node-1 snmpd[2059]: error on subcontainer 'ia_addr' insert (-1)
Jan 12 09:41:18 proxmox-node-1 corosync[2561]: error   [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:41:18 proxmox-node-1 corosync[2561]:  [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:41:19 proxmox-node-1 corosync[2561]: notice  [TOTEM ] Automatically recovered ring 0
Jan 12 09:41:19 proxmox-node-1 corosync[2561]:  [TOTEM ] Automatically recovered ring 0
Jan 12 09:41:35 proxmox-node-1 snmpd[2059]: error on subcontainer 'ia_addr' insert (-1)
Jan 12 09:41:40 proxmox-node-1 corosync[2561]: error   [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:41:40 proxmox-node-1 corosync[2561]:  [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:41:41 proxmox-node-1 corosync[2561]: notice  [TOTEM ] Automatically recovered ring 0
Jan 12 09:41:41 proxmox-node-1 corosync[2561]:  [TOTEM ] Automatically recovered ring 0
Jan 12 09:42:00 proxmox-node-1 systemd[1]: Starting Proxmox VE replication runner...
Jan 12 09:42:00 proxmox-node-1 systemd[1]: Started Proxmox VE replication runner.
Jan 12 09:42:05 proxmox-node-1 snmpd[2059]: error on subcontainer 'ia_addr' insert (-1)
Jan 12 09:42:07 proxmox-node-1 corosync[2561]: error   [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:42:07 proxmox-node-1 corosync[2561]:  [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:42:10 proxmox-node-1 corosync[2561]: notice  [TOTEM ] Automatically recovered ring 0
Jan 12 09:42:10 proxmox-node-1 corosync[2561]:  [TOTEM ] Automatically recovered ring 0
Jan 12 09:42:35 proxmox-node-1 snmpd[2059]: error on subcontainer 'ia_addr' insert (-1)
Jan 12 09:42:38 proxmox-node-1 corosync[2561]: error   [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:42:38 proxmox-node-1 corosync[2561]:  [TOTEM ] Marking ringid 0 interface 192.168.0.2 FAULTY
Jan 12 09:42:40 proxmox-node-1 corosync[2561]: notice  [TOTEM ] Automatically recovered ring 0
Jan 12 09:42:40 proxmox-node-1 corosync[2561]:  [TOTEM ] Automatically recovered ring 0

There is a issue only with external card communication, the ring buffer 0. Another one is good, I mean the communication via eth1 and ring buffer 1.

Look that I'm using Unicast traffic, I don't really know is it that an issue?

The last one is the errors which I'm seeing but I really don't know why. I think these variables is pushed to the hosts via cluster configuration..

Code: 
Jan 12 09:37:49 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 15) - errors in rule parameters: IN ACCEPT -source qnap_storage -dest +nodes_ipset_external -p tcp -dport 3493 # Allow NUT to QNAP storage
Jan 12 09:37:49 proxmox-node-1 pveproxy[2766]:   dest: no such ipset 'nodes_ipset_external'
Jan 12 09:37:49 proxmox-node-1 pveproxy[2766]:   source: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 3) - errors in rule parameters: IN ACCEPT -dest +nodes_ipset_external -p udp -dport 5404:5405 # Allow Corosync to external network card
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 4) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -p udp -dport 5404:5405 # Allow Corosync to external network card
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 6) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 32764:32769 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 7) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 32764:32769 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 8) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 2049 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 9) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 2049 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 10) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 111 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 11) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 111 # Allow NFS to the QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 12) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p udp -dport 3493 # Allow NUT to QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 13) - errors in rule parameters: IN ACCEPT -source +nodes_ipset_external -dest qnap_storage -p tcp -dport 3493 # Allow NUT to QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 14) - errors in rule parameters: IN ACCEPT -source qnap_storage -dest +nodes_ipset_external -p udp -dport 3493 # Allow NUT to QNAP storage
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   source: no such alias 'qnap_storage'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]:   dest: no such ipset 'nodes_ipset_external'
Jan 12 09:37:52 proxmox-node-1 pveproxy[2766]: /etc/pve/local/host.fw (line 15) - errors in rule parameters: IN ACCEPT -source qnap_storage -dest +nodes_ipset_external -p tcp -dport 3493 # Allow NUT to QNAP storage
 
Firewall settings are ok - looks like your cluster is not working correctly and therefore Datcenter firewall settings cannot be found be the node.
 
  • Like
Reactions: kristian.kirilov
So, I realized that anything works as I expected except that - the Proxmox host makes anti-blocking rules, so all traffic to SSH and 8006 are accepted, no matter firewall is enabled or not.

In this case the question is, how can I restrict access to this services only for particular IP addresses.
Can I use hosts.deny/allow or PAM to enable specific user to be able to login into the system only for these IP's?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back