Automatic change of firewall: blocks incoming ssh and http

[rty]

Abstract

My Proxmox Virtual Environments 8.[1,2] recently changed their behaviour. Things worked a few weeks ago and don't work now. I tracked the problem down to the firewalls and was able to solve it by disabling the firewalls for now.

I wonder

1. Should I worry about netstat reporting that port 8006 is open for tcp6 only?
2. Why do both firewalls block ssh and http? I never configured such a thing.
3. What is the proper configuration of both firewalls?

A qualified recommendation would be most appreciated!

Methods and results

Cluster with 2 nodes.

• Node 1 (pveversion 8.2.7): no ssh to node, no web access.
• Node 2 (pveversion 8.1.4): ssh to node works, no web access.
• Connections time out.
• On both nodes ssh localhost works.
• VMs and CTs are up and can be reached with ssh. However, they cannot open an ssh connection to the hosts.
• On both nodes sshd and pveproxy are up and ready (systemctl status).
• On both nodes netstat -tulpn state
  tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 835/sshd: /usr/sbin and
  tcp6 0 0 :::8006 :::* LISTEN 1616/pveproxy

The journal on node 1 reports that the firewall blocked both connection attempts:

Nov 09 19:25:28 pve-1 kernel: [UFW BLOCK] IN=vmbr0 OUT= MAC=34:64:a9:9a:5c:48:80:ee:73:83:60:32:08:00 SRC=172.22.2.2 DST=172.22.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=64871 DF PROTO=TCP SPT=55256 DPT=8006 WINDOW=64240 RES=0x00 SYN URGP=0
Nov 09 19:28:34 pve-1 kernel: [UFW BLOCK] IN=vmbr0 OUT= MAC=34:64:a9:9a:5c:48:80:ee:73:83:60:32:08:00 SRC=172.22.2.2 DST=172.22.2.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=17846 DF PROTO=TCP SPT=56084 DPT=8006 WINDOW=64240 RES=0x00 SYN URGP=0

systemctl stop pve-firewall solves the problem for node 2 (pve 8.2). For node 1 (pve 8.2), it does not. The service is down but ufw reports it is active.

Spoiler: status pve-firewall

○ pve-firewall.service - Proxmox VE firewall
Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled)
Active: inactive (dead) since Sat 2024-11-09 19:03:27 CET; 33min ago
Duration: 31min 29.546s
Process: 1182 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS)
Process: 1184 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS)
Process: 1185 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS)
Process: 1187 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS)
Process: 18514 ExecStop=/usr/sbin/pve-firewall stop (code=exited, status=0/SUCCESS)
Main PID: 1203 (code=exited, status=0/SUCCESS)
CPU: 17.362s

Nov 09 18:31:56 pve-1 pve-firewall[1203]: starting server
Nov 09 18:31:56 pve-1 systemd[1]: Started pve-firewall.service - Proxmox VE firewall.
Nov 09 19:03:25 pve-1 systemd[1]: Stopping pve-firewall.service - Proxmox VE firewall...
Nov 09 19:03:26 pve-1 pve-firewall[1203]: received signal TERM
Nov 09 19:03:26 pve-1 pve-firewall[1203]: server shutting down
Nov 09 19:03:26 pve-1 pve-firewall[1203]: clear PVE-generated firewall rules
Nov 09 19:03:26 pve-1 pve-firewall[1203]: server stopped
Nov 09 19:03:27 pve-1 systemd[1]: pve-firewall.service: Deactivated successfully.
Nov 09 19:03:27 pve-1 systemd[1]: Stopped pve-firewall.service - Proxmox VE firewall.
Nov 09 19:03:27 pve-1 systemd[1]: pve-firewall.service: Consumed 17.362s CPU time.

Spoiler: ufw status

Status: active

To Action From
-- ------ ----
12347/tcp ALLOW Anywhere
12347/tcp (v6) ALLOW Anywhere (v6)

For node 1 ufw allow ssh && ufw allow 8006 solves the problem.

Spoiler: pveversion --verbose on node 1

proxmox-ve: 8.2.0 (running kernel: 6.8.12-3-pve)
pve-manager: 8.2.7 (running version: 8.2.7/3e0176e6bb2ade3b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-3
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.5.13-6-pve-signed: 6.5.13-6
proxmox-kernel-6.5: 6.5.13-6
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.8
libpve-cluster-perl: 8.0.8
libpve-common-perl: 8.2.5
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.10
libpve-storage-perl: 8.2.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-4
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.4
pve-cluster: 8.0.8
pve-container: 5.2.0
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.0.7
pve-firmware: 3.14-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.4
pve-qemu-kvm: 9.0.2-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

Spoiler: pveversion --verbose on node 2

proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.1
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

 

[_gabriel]

For node 1 ufw allow ssh && ufw allow 8006 solves the problem.

UFW is not installed in Proxmox