pfSense and IPv6 CARP: vmbr0 not forwarding packets

[Cyan]

Hi Everyone,

I have a pair of pfSense firewalls running on the same Proxmox 6.0-6 VE node that seem to lose IPv6 multicast packets after a while. pfSense sends packets to ff02::12: that look like this:

06:48:29.896428 IP6 fe80::9c9f:5fff:fee0:8211 > ff02::12: ip-proto-112 36
06:48:30.164147 IP6 fe80::7c96:1cff:fe48:7496 > ff02::12: ip-proto-112 36

This works for a while, but after some time (less than about ten minutes), Proxmox stops forwarding packets to the VM's. The above was a tcpdump capture from the vmbr0 interface in Proxmox, so Proxmox is receiving those packets from pfSense. Now let's look at the VM interfaces individually:

root@prx1:~# tcpdump -i tap109i0 ip6 proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap109i0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:50:11.195225 IP6 fe80::9c9f:5fff:fee0:8211 > ff02::12: ip-proto-112 36
06:50:12.225846 IP6 fe80::9c9f:5fff:fee0:8211 > ff02::12: ip-proto-112 36
06:50:13.260955 IP6 fe80::9c9f:5fff:fee0:8211 > ff02::12: ip-proto-112 36

The same happens for the other VM too:

root@prx1:~# tcpdump -i tap110i0 ip6 proto 112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap110i0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:50:42.548082 IP6 fe80::7c96:1cff:fe48:7496 > ff02::12: ip-proto-112 36
06:50:43.961475 IP6 fe80::7c96:1cff:fe48:7496 > ff02::12: ip-proto-112 36
06:50:45.375199 IP6 fe80::7c96:1cff:fe48:7496 > ff02::12: ip-proto-112 36

After rebooting pfSense, each VM is able to see each other's multicast packets again.

I don't think this is a switch issue since these VM's are on the same Proxmox node, so the packets never get to the switch. It doesn't appear to be a pfSense issue either since the packets are making it to the Proxmox vmbr0 bridge (tcpdump above shows the packets being received). So it seems like something is preventing Proxmox from forwarding those packets on. ip6tables-save is empty. Changing to different VLAN's doesn't change the result either.

These same pfSense firewalls are running IPv4 based CARP without a problem, so I'm mystified about what the problem could be.

 

[Stoiko Ivanov]

For completeness sake (this issue was addressed in enterprise support) - disabling multicast_snooping and the multicast_querier:

echo 0 > /sys/class/net/vmbr0/bridge/multicast_router
echo 0 > /sys/class/net/vmbr0/bridge/multicast_snooping

seems to have fixed this issue