Hello,
I am running a PVE cluster with 3 nodes. Each node has a statically configured IPv6 address. When I set it up, I was able to ping across the cluster to another node from the WebUI host console no problem. But today when I was doing maintenance, I noticed that I couldn't ping anything on the same subnet besides the router. Running tcpdumps, I found that the pinging host would generate a NS and send it to a multicast address. I would see that appear on the receiving hosts tcpdump then no response would be made. Now while doing those captures, I saw that from time to time NS requests would arrive with unicast destinations and would be processed and a response would go back out from the host.
Now, when turning off the DC firewall, everything would start working again. So obviously a firewall issue. Though during my debugging I had a rule that any ipv6-icmp on the in chain would be allowed. Though that doesn't seem to be honored. NDP on the host FW is enabled. Setting logging wherever I could, I couldn't see any reason why it would be blocking.
Another weird thing is that I have another PVE instance with all the same package versions and can't replicate the issue easily.
Any help in figuring this out would be appreciated.
cluster.fw
host.fw
pveversion -v
/etc/network/interfaces
I am running a PVE cluster with 3 nodes. Each node has a statically configured IPv6 address. When I set it up, I was able to ping across the cluster to another node from the WebUI host console no problem. But today when I was doing maintenance, I noticed that I couldn't ping anything on the same subnet besides the router. Running tcpdumps, I found that the pinging host would generate a NS and send it to a multicast address. I would see that appear on the receiving hosts tcpdump then no response would be made. Now while doing those captures, I saw that from time to time NS requests would arrive with unicast destinations and would be processed and a response would go back out from the host.
Now, when turning off the DC firewall, everything would start working again. So obviously a firewall issue. Though during my debugging I had a rule that any ipv6-icmp on the in chain would be allowed. Though that doesn't seem to be honored. NDP on the host FW is enabled. Setting logging wherever I could, I couldn't see any reason why it would be blocking.
Another weird thing is that I have another PVE instance with all the same package versions and can't replicate the issue easily.
Any help in figuring this out would be appreciated.
cluster.fw
Code:
[OPTIONS]
enable: 1
policy_forward: ACCEPT
[IPSET hv-net]
<xx>.<x>.<x>.<x>/24
<xx>.<x>.<x>.<x>/24
<xx>.<x>.<x>.<x>/25
<xx>.<x>.<x>.<xxx>/25
<xx>.<x>.<x>.<x>/24
<xxxx>:<xxxx>:0:1::/64
<xxxx>:<xxxx>::/64
[IPSET rfc1918]
10.0.0.0/8
172.16.0.0/12
192.168.0.0/16
<xxxx>:<xxxx>::/40
<xxxx>:<xxxx>:<xxxx>:<xxxx>::/64
<xxxx>:<xxxx>:<xxxx>::/48
[RULES]
IN ACCEPT -p ipv6-icmp -log nolog
IN ACCEPT -p icmp -log nolog
IN ACCEPT -i vmbr0.25 -source +dc/hv-net -dest +dc/hv-net -p udp -dport 4789 -log nolog
IN BGP(ACCEPT) -i vmbr0.25 -source +dc/hv-net -dest +dc/hv-net -log nolog
IN ACCEPT -i vmbr0.25 -source +dc/hv-net -dest +dc/hv-net -p udp -dport 3784 -log nolog # BFD
IN ACCEPT -i vmbr0.11 -source +dc/hv-net -dest +dc/hv-net -p tcp -dport 60000:60050 -log nolog
IN SSH(ACCEPT) -i vmbr0.11 -source +dc/hv-net -dest +dc/hv-net -log nolog
IN Ceph(ACCEPT) -i vmbr0.12 -source +dc/hv-net -dest +dc/hv-net -log nolog
IN Ceph(ACCEPT) -i vmbr0.11 -source +dc/hv-net -dest +dc/hv-net -log nolog
IN SSH(ACCEPT) -i vmbr0.10 -log nolog
IN ACCEPT -i vmbr0.10 -source +dc/rfc1918 -dest +dc/hv-net -p tcp -dport 8006 -log nolog
IN ACCEPT -i mgmtbr0 -source +dc/rfc1918 -dest +dc/hv-net -p tcp -dport 8006 -log nolog
[group dns-srv]
IN DNS(ACCEPT) -source +dc/rfc1918 -log nolog
[group mysql-srv]
IN MySQL(ACCEPT) -source +dc/rfc1918 -log nolog
[group pg-srv]
IN PostgreSQL(ACCEPT) -source +dc/rfc1918 -log nolog
[group ping]
IN ACCEPT -p ipv6-icmp -log nolog
IN ACCEPT -p icmp -log nolog
[group ssh-srv]
IN SSH(ACCEPT) -source +dc/rfc1918 -log nolog
[group web-srv]
IN HTTPS(ACCEPT) -source +dc/rfc1918 -log nolog
IN HTTP(ACCEPT) -source +dc/rfc1918 -log nologhost.fw
Code:
[OPTIONS]
log_level_in: nolog
ndp: 1
nftables: 1pveversion -v
Code:
proxmox-ve: 9.2.0 (running kernel: 7.0.14-17-pve)
pve-manager: 9.2.20 (running version: 9.2.20/49318c671b82f31e)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-17-pve-signed: 7.0.14-17
proxmox-kernel-7.0: 7.0.14-17
proxmox-kernel-7.0.14-16-pve-signed: 7.0.14-16
proxmox-kernel-6.17: 6.17.13-21
proxmox-kernel-6.17.13-21-pve-signed: 6.17.13-21
proxmox-kernel-6.14: 6.14.11-9
proxmox-kernel-6.14.11-9-pve-signed: 6.14.11-9
ceph: 19.2.6-pve4
ceph-fuse: 19.2.6-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
dnsmasq: 2.91-1+deb13u2
frr-pythontools: 10.6.1-1+pve3
ifupdown2: 3.3.0-1+pmx12
intel-microcode: 3.20251111.1~deb13u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.3
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.2
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.2
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
proxmox-backup-client: 4.2.6-1
proxmox-backup-file-restore: 4.2.6-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.8
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.11
pve-edk2-firmware: 4.2026.08-1
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.6
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.8
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1/etc/network/interfaces
Code:
auto lo
iface lo inet loopback
auto enp4s0f0
iface enp4s0f0 inet manual
auto eno1
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
auto enp4s0f1
iface enp4s0f1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp4s0f0 enp4s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000
auto vmbr0.11
iface vmbr0.11 inet static
address <xx>.<x>.<x>.<xx>/25
auto vmbr0.12
iface vmbr0.12 inet static
address <xx>.<x>.<x>.<xxx>/25
auto mgmtbr0
iface mgmtbr0 inet static
address <xx>.<x>.<x>.<xx>/24
gateway <xx>.<x>.<x>.<x>
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
iface mgmtbr0 inet6 static
address <xxxx>:<xxxx>::<xxxx>:<xxxx>:<xxxx>:<xxxx>/64
gateway <xxxx>:<xxxx>::<x>
auto vmbr0.10
iface vmbr0.10 inet static
address <xx>.<x>.<x>.<xx>/24
mtu 1500
auto vmbr0.25
iface vmbr0.25 inet static
address <xx>.<x>.<x>.<xx>/24
source /etc/network/interfaces.d/*