OPNsense best practices?

H4R0 · Feb 21, 2021

Looks good to me.

Sadly I have no documentation for the ha part of opnsense and it's been over a year since.

I would make sure the multicast traffic works.

Install tcpdump on both pve nodes and run "tcpdump -i <interface> -s0 -vv net 224.0.0.0/4"

Make sure multicast traffic from firewall1 leaves the interface on node1 and arrives on node2.

Dunuin · Feb 21, 2021

H4R0 said:
I would make sure the multicast traffic works.

Install tcpdump on both pve nodes and run "tcpdump -i <interface> -s0 -vv net 224.0.0.0/4"

Make sure multicast traffic from firewall1 leaves the interface on node1 and arrives on node2.

I think I have seen some firewall logs blocking IGMP and 224.0.0.0/4. I will check that again.

Edit:
Looks like OPNsense is blocking Multicast

I disabled the Firewalls on Datacenter level and allowed IGMP on OPNsense on all interfaces and allowed 224.0.0.0/4 on OPNsense on all interfaces but both are still master.

Edit:
Run on Proxmox Hypervisor (who hosts the Master OPNsense VM):

Code:

tcpdump -i bond0 -s0 -vv net 224.0.0.0/4
tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 by                                                                                                                                                             tes
19:19:16.803580 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (1                                                                                                                                                             12), length 56)
    OPNsense2.localdomain > vrrp.mcast.net: vrrp OPNsense2.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 147.71.3.20,93.250.81.92,pfiber2.physics.metu.edu.tr,231.225.190.165,28.134.249.124,33.17.203.187,228.244.37.117
19:19:16.803583 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): ec2-54-68-25-186.us-west-2.compute.amazonaws.com,165.208.42.80,133.93.73.1,175.159.5.251,140.150.107.180,222.90.245.41,79.143.211.179
19:19:16.803588 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.3 > vrrp.mcast.net: vrrp 192.168.42.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 245.188.131.107,16.13.40.215,197.15.40.176,host31-48-241-105.range31-48.btcentralplus.com,149.33.47.231,dynamic-078-055-223-231.78.55.pool.telefonica.de,108.137.210.209
19:19:17.521163 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.43.2 > vrrp.mcast.net: vrrp 192.168.43.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 147.71.3.20,93.250.81.93,58.235.201.79,149.144.147.233,66.86.128.173,156.147.204.101,231.127.122.15
19:19:17.521218 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): ec2-54-68-25-186.us-west-2.compute.amazonaws.com,165.208.42.81,softbank060140210245.bbtec.net,106.55.139.138,246.146.203.78,ns3139586.ip-79-137-68.eu,nts-116.71-182-65.nts-online.net
19:19:17.521245 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.2 > vrrp.mcast.net: vrrp 192.168.42.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36, addrs(7): 245.188.131.107,16.13.40.216,255.223.229.20,230.217.245.111,7.144.24.165,197.200.18.104,127.83.10.227
19:19:18.232783 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.3 > vrrp.mcast.net: vrrp 192.168.42.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): host-79-1-77-35.business.telecomitalia.it,139.202.212.98,205.175.6.125,250.72.218.130,189-71-45-68.user.veloxzone.com.br,dns224.online.tj.cn,37.83.226.87
19:19:30.877110 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.2 > vrrp.mcast.net: vrrp 192.168.42.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36, addrs(7): 63.0.56.143,213.178.46.37,116.171.171.196,107.111.40.166,196.105.235.224,ip-174-151-171-217.nsvltn.spcsdns.net,239.167.55.28
19:19:44.063748 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    OPNsense2.localdomain > vrrp.mcast.net: vrrp OPNsense2.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 183.54.58.208,43.15.153.217,22.38.14.218,148.123.108.228,142.223.176.223,160.101.28.82,171.155.248.170
19:19:47.314667 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): n119237231114.netvigator.com,p1013-ipad06daianji.nara.ocn.ne.jp,39.26.2.156,199.166.81.119,184-092-099-233.res.spectrum.com,147.246.112.32,215.2.80.176
19:19:55.406792 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.3 > vrrp.mcast.net: vrrp 192.168.42.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 180.77.23.111,pool-109-191-181-174.is74.ru,127.3.253.72,191.75.29.179,165.28.97.224,17.242.92.161,p366180-ipngn200407daianjibetu.nara.ocn.ne.jp
19:19:58.674655 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.43.2 > vrrp.mcast.net: vrrp 192.168.43.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 62.175.138.171.static.user.ono.com,241.110.219.136,234.231.92.86,107.46.113.81,229.191.236.83,74.red-2-140-66.dynamicip.rima-tde.net,244.28.128.216
19:20:00.748783 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.43.2 > vrrp.mcast.net: vrrp 192.168.43.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 252.100.3.231,232.87.39.9,softbank126203049209.bbtec.net,p5787baa2.dip0.t-ipconnect.de,c-50-178-38-132.hsd1.in.comcast.net,hs622-17.houston.sc.hpecorp.net,224.17.238.95
19:20:13.923264 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    OPNsense2.localdomain > vrrp.mcast.net: vrrp OPNsense2.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): p2406229-ipngn16301hodogaya.kanagawa.ocn.ne.jp,117.72.158.206,ec2-52-50-169-252.eu-west-1.compute.amazonaws.com,aut75-1-81-57-118-228.fbx.proxad.net,c-73-139-179-247.hsd1.fl.comcast.net,132.sub-70-197-216.myvzw.com,234.83.157.75
19:20:21.088989 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.3 > vrrp.mcast.net: vrrp 192.168.42.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): bbb53563.virtua.com.br,244.207.17.247,185.248.167.140,78.185.97.209.dynamic.ttnet.com.tr,225.149.89.163,61.243.183.9,41.190.179.177
19:20:22.508513 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    OPNsense2.localdomain > vrrp.mcast.net: vrrp OPNsense2.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 180.149.220.192,195.64.104.55,140-127-216-118.nuk.edu.tw,c-73-85-7-61.hsd1.fl.comcast.net,131.150.96.177,148.99.198.147,15.73.182.209
19:20:32.493109 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.42.3 > vrrp.mcast.net: vrrp 192.168.42.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 238.211.8.165,49.97.139.75,26.172.55.14,201.red-83-48-176.dynamicip.rima-tde.net,136.136.54.84,209.160.89.81,125.235.246.215.adsl.viettel.vn
19:20:35.957671 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
^C    192.168.43.2 > vrrp.mcast.net: vrrp 192.168.43.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 230.246.156.218,202.29.133.71,170.171.196.5,50.49.47.90,ip-200-13-31-49-mx.marcatel.net.mx,151.215.202.141,64.14.180.142

18 packets captured
632 packets received by filter
608 packets dropped by kernel

"OPNsense2.localdomain" is the Backup OPNsense-VM on the FreeNAS host.

H4R0 · Feb 21, 2021

Dunuin said:
18 packets captured
632 packets received by filter
608 packets dropped by kernel

Almost all packets are dropped ?

Does the output of "ifconfig bond0" state the drops as well ?

I only have 289 drops for 200GB of traffic and if I run tcpdump there are none.

Do you have the firewall of pve enabled or custom iptable rules ?

"iptables -L"

Dunuin · Feb 21, 2021

Does the output of "ifconfig bond0" state the drops as well ?

Code:

ifconfig bond0
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 9000
        ether xx:xx:xx:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 673070632  bytes 251598080684 (234.3 GiB)
        RX errors 0  dropped 166650  overruns 670294  frame 0
        TX packets 615745933  bytes 1878038477278 (1.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

H4R0 said:
Do you have the firewall of pve enabled or custom iptable rules ?

Normally pve firewalls are enabled. But I captured tcpdumb after disableing the firewall at datacenter level.

H4R0 said:
"iptables -L"

Code:

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I thought it might be something because bond0 is vlan tagged. I also run tcpdump on the WAN bridge (vmbr2) what should be untagged but it looks the same:

Code:

tcpdump -i vmbr2 -s0 -vv net 224.0.0.0/4
tcpdump: listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
19:43:45.341526 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 246.qarestr.sub-166-180-63.myvzw.com,106.232.193.224,c-76-97-165-193.hsd1.ga.comcast.net,186.20.200.142,c-50-168-196-81.hsd1.ut.comcast.net,253.195.152.126,122.186-69-176.uio.satnet.net
19:43:45.462756 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 52.170.58.16,ip72-208-196-98.ph.ph.cox.net,234.60.70.127,195.143.123.15,1F2EC052.dsl.pool.telekom.hu,169.165.81.193,247.227.119.41
19:43:46.374662 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 52.170.58.16,ip72-208-196-99.ph.ph.cox.net,164.113.162.122,124x35x18x150.ap124.ftth.ucom.ne.jp,56.82.231.57,56.244.23.52,90.185.19.194
19:43:46.883274 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): c-76-102-229-28.hsd1.ca.comcast.net,25.148.81.237,45.75.136.37,228.235.1.24,81-175-250-242.co.dnainternet.fi,dslb-092-075-206-248.092.075.pools.vodafone-ip.de,44.31.105.244
19:43:47.406081 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): c-76-102-229-28.hsd1.ca.comcast.net,25.148.81.238,nothing.attdns.com,33.46.179.46,183.34.151.80,98.ba.c1ad.ip4.static.sl-reverse.com,27.184.79.250
19:43:48.307852 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): dhcp-077-248-186-213.chello.nl,ip-72-60-179-83.hrsnnj.spcsdns.net,100.211.136.23,232.53.49.182,120.196.144.76,131.251.27.78,101.91.57.154
19:43:48.460925 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): dhcp-077-248-186-213.chello.nl,ip-72-60-179-84.hrsnnj.spcsdns.net,ppp141255015180.access.hol.gr,159.251.236.184,52.183.96.245,15.129.143.62,120.241.78.37
19:43:51.151978 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): cpe-24-163-10-116.triad.res.rr.com,44.86.232.26,189-72-64-139.ccoce700.dsl.brasiltelecom.net.br,230.83.99.45,245.76.235.172,ec2-52-78-150-94.ap-northeast-2.compute.amazonaws.com,softbank060144221100.bbtec.net
19:43:53.651631 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 28.67.154.85,130.178.97.99,14.25.168.115,238.101.169.15,32.195.35.194,78-23-88-46.access.telenet.be,236.219.12.56
19:43:58.800486 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 4.169.77.106,223.48.152.136,244.3.61.225,229.206.223.225,1.45.180.175,111.120.126.49,246.180.31.171
19:44:01.874519 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 146.203.16.62,mobile-166-173-118-183.mycingular.net,198-144-240-10.take2hosting.com,245.230.173.242,247.53.204.33,103.238.166.215,123.179.154.233
19:44:09.757608 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 94.48.138.18,171.87.0.95,c211-28-205-74.chirn2.vic.optusnet.com.au,85.102.5.25.dynamic.ttnet.com.tr,102.6.137.66,105.120.228.86,155.10.72.232
19:44:15.405867 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): ec2-18-200-251-23.eu-west-1.compute.amazonaws.com,26.236.44.186,77-165-53-82.fixed.kpn.net,254.76.124.85,48.181.218.2,56.187.43.23,25.59.209.0
19:44:36.801424 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 134.149.150.38,55.15.79.13,host196.181-105-135.telecom.net.ar,1.3.232.20,244.65.234.237,host-55C6E726.sileman.net.pl,38.228.13.204
19:44:41.089888 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 8.171.132.171,104-189-229-27.lightspeed.sntcca.sbcglobal.net,177-172-14-24.user.vivozap.com.br,250.185.83.177,ec2-13-211-176-70.ap-southeast-2.compute.amazonaws.com,247-174-235-166.mobile.uscc.net,130.148.229.252
19:44:42.531237 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 20.164.156.91,167.247.2.86,.,221.198.65.255,172.210.224.4,243.70.189.56,206.246.102.218.nni.net
19:44:45.328608 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
^C    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 33.77.11.179,225.40.187.39,mobile-107-233-90-184.mycingular.net,232.100.213.127,101.158.87.192,ip-184-250-101-224.sanjca.spcsdns.net,100.123.181.239

17 packets captured
196 packets received by filter
173 packets dropped by kernel

Not sure how to interpret the tcp packets in detail but 192.168.0.2 and 192.168.0.3 are the physical WAN IPs of the two OPNsense VM so it looks like the switch is routing the packets between the hosts?

This is the tcpdump of the WAN bridge on the other host (FreeNAS running the Backup OPNsense VM):

Code:

tcpdump -i bridge2 -s0 -vv net 224.0.0.0/4
tcpdump: listening on bridge2, link-type EN10MB (Ethernet), capture size 262144 bytes
20:04:43.208219 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 67.200.16.140,98.119.182.203,49.69.128.216,101.130.40.61,135.108.51.217,47-211-14-183.alexcmtk01.res.dyn.suddenlink.net,128.151.80.210
20:04:44.633952 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): c-73-1-52-210.hsd1.fl.comcast.net,142-105-73-67.biz.spectrum.com,155.199.126.74,40.239.30.91,143.237.202.113,HSI-KBW-37-49-71-112.hsi14.kabel-badenwuerttemberg.de,85.97.173.71.dynamic.ttnet.com.tr
20:04:46.042858 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 117.246.137.206,20.47.205.41,4.209.118.10,205.36.233.163,110.sub-97-205-113.myvzw.com,105.177.183.159,47.156.157.45
20:04:47.518876 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 155.96.139.94,pool-71-97-199-58.hstntx.dsl-w.verizon.net,softbank060101253052.bbtec.net,183.177.62.69,199.221.116.194,250.75.182.192,18.30.234.153
20:04:47.658155 IP (tos 0xc0, ttl 1, id 13337, offset 0, flags [DF], proto IGMP (2), length 36, options (RA))
    192.168.0.1 > all-systems.mcast.net: igmp query v3
20:04:48.946765 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 123.65.89.178,a23-45-7-210.deploy.static.akamaitechnologies.com,254.219.46.19,55.178.40.137,160.166.255.18,26.144.103.58,52.190.42.151
20:04:50.152067 IP (tos 0x0, ttl 4, id 40276, offset 0, flags [none], proto UDP (17), length 151)
    192.168.0.1.56790 > 239.255.255.250.1900: [udp sum ok] UDP, length 123
20:04:50.430632 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 237.97.214.131,56.140.63.6,dsl-233-121.iae.nl,255.231.189.229,182.206.136.31,221x252x32x92.ap221.ftth.ucom.ne.jp,142.141.101.110
20:04:51.859270 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 61.237.109.84,50-244-48-153-static.hfc.comcastbusiness.net,223.208.114.226,36.85.12.129,c-73-234-20-249.hsd1.ma.comcast.net,19.55.76.30,218.55.168.188
20:04:53.266712 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 110.107.133.178,245.219.141.151,177.166.215.139.adsl-pool.jlccptt.net.cn,247.24.32.78,33.9.203.40,64.129.250.80,134.149.162.173
20:04:54.691020 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 229.114.210.169,202.109.235.243,172.18.106.240,12.207.131.72,101.94.54.180,158.99.55.218,42.97.68.91.rev.sfr.net
20:04:55.152020 IP (tos 0x0, ttl 4, id 40304, offset 0, flags [none], proto UDP (17), length 151)
    192.168.0.1.56790 > 239.255.255.250.1900: [udp sum ok] UDP, length 123
20:04:56.114358 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 63.65.174.68,51.124.23.114,143.119.19.137,3.204.145.76,136-55-102-188.googlefiber.net,5.183.234.235,162.102.26.217
20:04:57.543773 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 196.17.221.133,111.66.235.165,48.129.148.145,214.41.146.28,110.152.107.93,024-217-099-066.res.spectrum.com,152.153.128.208
20:04:59.019482 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): rrcs-67-52-4-247.west.biz.rr.com,100.161.136.102,123.170.24.31,19.19.1.45,95-54-58-143.dynamic.lenobl.dslavangard.ru,static-host-66-18-36-180.epbinternet.com,250.53.146.181
20:05:00.152295 IP (tos 0x0, ttl 4, id 41348, offset 0, flags [none], proto UDP (17), length 151)
    192.168.0.1.56790 > 239.255.255.250.1900: [udp sum ok] UDP, length 123
20:05:00.437954 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 191-214-159-255.user3g.veloxzone.com.br,235.176.206.31,107.69.1.244,59.68.62.100,239.152.30.154,ec2-3-1-215-254.ap-southeast-1.compute.amazonaws.com,99-194-25-32.dyn.centurytel.net
20:05:01.861372 IP (tos 0xe0, ttl 255, id 0, offset 0, flags [DF], proto VRRP (112), length 56)
^C    192.168.0.3 > vrrp.mcast.net: vrrp 192.168.0.3 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): nthygo058132.hygo.nt.ngn.ppp.infoweb.ne.jp,23.120.250.104,23-120-250-104.lightspeed.sntcca.sbcglobal.net,221.122.148.179,62.219.166.123,bzq-166-123.dsl.bezeqint.net,82.51.123.219

18 packets captured
12694 packets received by filter
0 packets dropped by kernel

Dunuin · Feb 21, 2021

So looks like the Proxmox server recieves CARP packets of the FreeNAS server but the FreeNAS server not CARP packets of the Proxmox server?

Edit:
"packets dropped by kernel" is caused if tcp dump cant parse the packets fast enough. If I add "-B 16000" to the tcpdump options to increase the buffer to 16kb no packets will be dropped by the kernel.

Edit:
Looks like CARP packets from my Proxmox host are recieved on the FreeNAS host. The tcpdump command just doesn't filter right on FreeBSD. If I run this on the FreeNAS host I can see some CARP packages:

Code:

tcpdump -B 16000 -i lagg0 -s0 -vv | grep VRRPv2
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
    OPNsense.localdomain > vrrp.mcast.net: vrrp OPNsense.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 25.25.152.83,164.253.14.54,16.135.203.78,38.95.41.43,52.129.195.246,215.228.238.228,196.227.244.105
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 165.166.115.211,254.244.252.54,adsl-99-21-102-11.dsl.sndg02.sbcglobal.net,15.245.236.85,245.135.157.236,117.245.187.204,bbs.marisfrolg.com
    192.168.42.2 > vrrp.mcast.net: vrrp 192.168.42.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36, addrs(7): 205.251.9.109,202.203.29.221,115.10.222.173,201.221.105.196,no-dns-yet.as25178.net,224.252.83.193,183.57.104.121
    192.168.0.2 > vrrp.mcast.net: vrrp 192.168.0.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 165.166.115.211,254.244.252.81,22.253.57.125,156.65.164.15,151.219.97.147,255.160.175.215,164.121.231.229
    192.168.42.2 > vrrp.mcast.net: vrrp 192.168.42.2 > vrrp.mcast.net: VRRPv2, Advertisement, vrid 5, prio 0, authtype none, intvl 1s, length 36, addrs(7): 205.251.9.109,202.203.29.248,210.100.10.105,87-231-148-54.rev.numericable.fr,44.110.97.158,3.168.136.87,147.143.215.96
    OPNsense.localdomain > vrrp.mcast.net: vrrp OPNsense.localdomain > vrrp.mcast.net: VRRPv2, Advertisement, vrid 3, prio 0, authtype none, intvl 1s, length 36, addrs(7): 25.25.152.83,164.253.14.81,static.vnpt.vn,28.61.204.31,202.191.76.27,145.143.EARLY-REGISTRATION.of.SURFnet.invalid,c-50-141-18-158.hsd1.ca.comcast.net

(Backup OPNsense was shutdown so its normal that there are no CARP packets from 192.168.0.3 found)

So the question is why is "cpdump -B 16000 -i lagg0 -s0 -vv | grep VRRPv2" showing me CARP packages from "192.168.42.2" and "tcpdump -i bridge2 -s0 -vv net 224.0.0.0/4" not?

Edit:
Looks like my lagg0 bond on the FreeNAS host is receiving CARP packages but I cant find any on the vlan2 or bridge2 interface.

Edit:
Network looks like this:
OPNsense VM -> vmbr2 (bridge) -> bond0.2 (vlan) -> bond0 (failover) -> two NICs -> Switch <- two NICs <- lagg0 (failover) <- vlan2 <- bridge2 <- OPNsense2 VM

If I tcpdump vmbr2 I see packets from both VMs.
If I tcpdump lagg0 I see packets from both VMs.
If I tcpdump vlan2 or bridge2 I see only packets from OPNsense2 (the VM on the FreeNAS host).

Any idea how CARP packets can pass vlan2 to lagg0 but get lost if traveling from lagg0 to vlan2?

Tcpdump vlan2:

Code:

tcpdump -B 16000 -i vlan2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 204.26.121.131,113.231.163.36,111.27.69.51,139.52.213.104,103.208.241.213,47.0.182.224,252.5.96.64
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 41.158.64.171,149.16.125.193,4.191.90.167,232.125.20.154,100.184.50.211,12.127.42.235,115.221.179.11
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 56.238.168.180,162.56.17.240,137.1.45.56,91.38.150.2,7.134.32.57,88.104.65.61,66.78.169.91

Tcpdump lagg0:

Code:

tcpdump -B 16000 -i lagg0 -s0 -vv -n | grep VRRPv2
tcpdump: listening on lagg0, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
    192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
    192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.20,142.113.126.246,185.154.196.161,103.0.53.151,212.222.211.243,85.170.90.42
    192.168.43.3 > 224.0.0.18: vrrp 192.168.43.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 3, prio 100, authtype none, intvl 1s, length 36, addrs(7): 96.66.4.97,132.211.112.69,212.103.17.121,143.149.122.238,4.30.250.155,65.48.107.187,65.63.152.79
    192.168.42.3 > 224.0.0.18: vrrp 192.168.42.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 5, prio 100, authtype none, intvl 1s, length 36, addrs(7): 32.109.10.203,100.5.167.83,103.19.183.233,178.227.26.106,226.114.149.25,181.206.12.219,83.59.18.64
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.21,135.66.16.237,211.4.211.42,117.193.93.14,212.76.235.95,104.10.216.75

Tcpdump vmbr2:

Code:

tcpdump -B 16000 -i vmbr2 -s0 -vv -n | grep VRRPv2
tcpdump: listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.87,70.65.213.181,67.105.250.170,222.9.115.236,50.252.143.196,169.224.5.94
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.88,58.219.37.12,15.210.142.91,192.209.83.22,41.242.4.18,137.13.56.178
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 121.117.48.10,56.7.253.89,16.104.17.98,215.206.131.78,42.128.191.9,19.252.151.219,186.215.188.18
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.99,48.110.165.149,172.62.82.112,114.186.137.221,186.29.255.226,22.90.25.111
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 75.106.105.61,145.109.17.100,86.107.71.132,149.44.192.90,221.144.124.56,111.128.79.123,117.35.199.20
    192.168.0.3 > 224.0.0.18: vrrp 192.168.0.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.39,133.190.110.130,88.215.243.94,211.204.175.144,249.115.200.64,153.255.214.192
    192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 35.220.254.150,215.66.139.40,151.189.205.255,16.22.81.156,215.5.36.88,254.222.251.113,140.137.102.211

Dunuin · Feb 22, 2021

I also checked the headers with wireshark.

The packets found on lagg0 got vlan 2 in header:

192.168.0.2 > 224.0.0.18: vrrp 192.168.0.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36, addrs(7): 111.100.125.40,62.223.145.21,135.66.16.237,211.4.211.42,117.193.93.14,212.76.235.95,104.10.216.75

But if I look on the "vlan2" interface, that is set to vlan 2, there are no packets found.

Dunuin · Feb 22, 2021

I think I know now whats going on...

I removed that lagg0 and bond0 and it looks like the ConnectX NIC on the FreeNAS host can't receive but send CARP packets. The other NIC (which was the passive one of the bond) is able to receive CARP packets. I thinks thats by I was able to see CARP packets on the lagg0 but not on the vlan2 interface. The lagg0 received CARP packets using the passive NIC and because they came from the passive NIC they were dropped because the active NIC (the ConnectX3) got priority.

So the question is why the ConnectX3 in the FreeNAS server can't receive but send CARP packets. I use the same model (mcx311a-xcat) in the Promox host and here both directions are working fine.

H4R0 · Feb 23, 2021

Dunuin said:
I think I know now whats going on...

I removed that lagg0 and bond0 and it looks like the ConnectX NIC on the FreeNAS host can't receive but send CARP packets. The other NIC (which was the passive one of the bond) is able to receive CARP packets. I thinks thats by I was able to see CARP packets on the lagg0 but not on the vlan2 interface. The lagg0 received CARP packets using the passive NIC and because they came from the passive NIC they were dropped because the active NIC (the ConnectX3) got priority.

So the question is why the ConnectX3 in the FreeNAS server can't receive but send CARP packets. I use the same model (mcx311a-xcat) in the Promox host and here both directions are working fine.

Try with the onboard intel/broadcom nic to verify it's not a driver issue with mellanox.

BSD always lacks behind and has weird issues. Might be some freenas setting but I doubt that.

You can also try to set the nic to promisc mode but unlikely it fixes it.

Dunuin · Feb 23, 2021

H4R0 said:
Try with the onboard intel/broadcom nic to verify it's not a driver issue with mellanox.

Ports on the Switch are setup identical. Both are using tagged VLAN. The onboard Intel NIC can send/receive CARP, the Mellanox can only send.

H4R0 said:
BSD always lacks behind and has weird issues. Might be some freenas setting but I doubt that.

ConnectX3 should work out of the box for FreeNAS. I googled it first before buying. Only ConnectX2 should be problematic. But maybe CARP isn't just the basic stuff other people are using on their NAS if they buy a ConnectX3.

H4R0 said:
You can also try to set the nic to promisc mode but unlikely it fixes it.

It was already enabled.

Dunuin · Feb 24, 2021

No one knows why that ConnectX3 isn't receiving CARP packets on the FreeNAS host. I asked in the OPNsense, FreeNAS, Mellanox, FreeBSD forums and here.

As a workaround I am now using the Gbit Intel NIC on the FreeNAS host for the backup OPNSense VM. It will be slow if the master OPNsense is offline because all 8 VLANs will be routed over a single Gbit NIC instead of the 10 Gbit NIC, like the master OPNsense, but atleast HA is working this way.

H4R0 · Feb 24, 2021

Dunuin said:
No one knows why that ConnectX3 isn't receiving CARP packets on the FreeNAS host. I asked in the OPNsense, FreeNAS, Mellanox, FreeBSD forums and here.

As a workaround I am now using the Gbit Intel NIC on the FreeNAS host for the backup OPNSense VM. It will be slow if the master OPNsense is offline because all 8 VLANs will be routed over a single Gbit NIC instead of the 10 Gbit NIC, like the master OPNsense, but atleast HA is working this way.

I don't see a problem, it serves as backup only and you should not route over vlans anyway. Subnet traffic never hits the gateway.

You can try to upgrade the x3 firmware https://www.mellanox.com/support/firmware/connectx3en

Also update freenas to truenas there are many changes in bsd kernel from 11.3 to 12.0.

Dunuin · Feb 24, 2021

H4R0 said:
I don't see a problem, it serves as backup only and you should not route over vlans anyway. Subnet traffic never hits the gateway.

And what would be better? I've got a lot of different groups of hosts that all need to be somehow connected so I need some kind of routing between them. For example:

LAN: only trusted hosts, so access to everything
DMZ: mainly for my VMs/LXCs. Should block any in/out by default except I allow needed ports
RETRO: for all my retro computers. Should be able to access the NAS (FTP) but without access to the internet
IOT: should be offline by default except I allow ports for individual hosts
GUEST: should be able to access the internet but not other subnets

H4R0 said:
You can try to upgrade the x3 firmware https://www.mellanox.com/support/firmware/connectx3en

Already did that without success.

H4R0 said:
Also update freenas to truenas there are many changes in bsd kernel from 11.3 to 12.0.

Right now I'm not even running the latest 11.3. Needed to roll back to 11.3U4.1 because of alot of bugs on 11.3U5. For example I wan't able to do any replications anymore. As long as they don't get 11.3 working without problems I don't want to risk a upgrade to 12.0. And there are so much big changes (switch to OpenZFS and so on) that I want to wait until it is more stable and better tested.

H4R0 · Feb 24, 2021

Dunuin said:
And what would be better? I've got a lot of different groups of hosts that all need to be somehow connected so I need some kind of routing between them. For example:

LAN: only trusted hosts, so access to everything
DMZ: mainly for my VMs/LXCs. Should block any in/out by default except I allow needed ports
RETRO: for all my retro computers. Should be able to access the NAS (FTP) but without access to the internet
IOT: should be offline by default except I allow ports for individual hosts
GUEST: should be able to access the internet but not other subnets

"It will be slow if the master OPNsense is offline because all 8 VLANs will be routed over a single Gbit"

For example if you have a storage server you add all needed vlans to it, so it can be reached over the subnet and not over the gateway.

If it's low traffic it's fine, just keep in mind the gateway has to route it and always create source/destination port firewall rules, otherwise segmentation doesn't make much sense. But in that case you should not have problems with your mentioned 1G.

Dunuin · Feb 24, 2021

H4R0 said:
"It will be slow if the master OPNsense is offline because all 8 VLANs will be routed over a single Gbit"

For example if you have a storage server you add all needed vlans to it, so it can be reached over the subnet and not over the gateway.

I already did this. And I also got two vlans (DMZNAS, LANNAS) especially for NAS access that are not connected to the OPNsense. That way my VMs can use 2 virtio NICs. For example one NIC with 1500 MTU to my DMZ vlan for internet access and one with 9000 MTU to my DMZNAS vlan to access the NAS. The NAS got a IP in most vlans.

H4R0 said:
If it's low traffic it's fine, just keep in mind the gateway has to route it and always create source/destination port firewall rules, otherwise segmentation doesn't make much sense. But in that you should not have problems with your mentioned 1G.

I will do that.

Ramalama · Feb 24, 2021

Just offtopic hint/question...

Why you don't use sr-iov?
sr-iov is there to virtualize your nic into multiple functions and pass them through to your guest.

So in the end you have as many 10GB/s nics as you want and you don't need to bother with virtio nics and the perfomance issues/bugs on freebsd with virtio.
(Almost every 10gb nic can do sr-iov)

Additional to that, your carp issue could come from, that the physical mellanox nic doesn't has mac adresses in the nics maddr table. (Yes physical nics have a mac address table too)

The only thing is, I don't know the commands on freebsd, but on linux you can add a macaddress with:
bridge fdb add XX:XX:XX:XX:XX:XX dev physical_nic_name
(The mac address is from the vm on the virtual bridge that uses the physical nic)

But really, in my opinion sr-iov is exactly made for passing through a nic to a guest, if you have that option, use it.

Cheers

Dunuin · Feb 24, 2021

Ramalama said:
Just offtopic hint/question...

Why you don't use sr-iov?
sr-iov is there to virtualize your nic into multiple functions and pass them through to your guest.

So in the end you have as many 10GB/s nics as you want and you don't need to bother with virtio nics and the perfomance issues/bugs on freebsd with virtio.
(Almost every 10gb nic can do sr-iov)

Additional to that, your carp issue could come from, that the physical mellanox nic doesn't has mac adresses in the nics maddr table. (Yes physical nics have a mac address table too)

The only thing is, I don't know the commands on freebsd, but on linux you can add a macaddress with:
bridge fdb add XX:XX:XX:XX:XX:XX dev physical_nic_name
(The mac address is from the vm on the virtual bridge that uses the physical nic)

But really, in my opinion sr-iov is exactly made for passing through a nic to a guest, if you have that option, use it.

Cheers

Thanks for the hint with the maddr table. If will check that.

The ConnectX3 is supporting SR-IOV but I can't activate it using the bootloader tool because it can't write any changes there. Same bug on all 3 NICs I got. Still after Update to latest firmware. It should be possible to enable it by dumping the configs, editing them and flashing them again but I dont wanted to brick something and wasn't able to find the right configs for my model.
The second problem is that it looks like my Supermicro X10SSL-F BIOS (my FreeNAS host) doesn't support SR-IOV. I wasn't able to find anything in the BIOS/Manual and only this in the FAQ:

Supermicro X10/X11 dual processors and multiple processors motherboards will support SR-IOV in supported Windows and Linux OS matrix chart. User can check Supermicro compatiblity chart for best practice.

Trying to deploy SR-IOV with X10 UP may suffer from failure of assignment due on ACS support inside Windows, this is due on limitation of the support from PEG and C220 series PCH.
For X10 UP, such as X10SL7-F/X10SLE-DF/X10SDD-16C-F/X10SDD-F/X10SLA-F/X10SLD-F/X10SLD-HF/X10SLE-F/X10SLE-HF/X10SLH-F/X10SLL+-F/X10SLL-S/X10SLL-SF/X10SLM+-F/X10SLM+-LN4F/X10SLM-F, the default LOM/MicroLP network controller of those boards is with i210 mostly. Intel i210 does not support SR-IOV feature. X10 UP PEG does not support SR-IOV, either.

And for HA it is required that both OPNsenses are using the same interface names. If I need to use virtio on the FreeNAS host because SR-IOV isn't available there, I can't use SR-IOV on the Proxmox host because one interface would be vtnetX and one mlxenX.

But yes, if SR-IOV would work on both hosts I would have prefered that.

Ramalama · Feb 25, 2021

Well yes, that's sad. I know sr-iov isn't possible always, just meant if you can, do it. But you can't, so shit happens

However another hint, on freebsd some virtio nic offloading features are broken. Like checksum offloading etc. Just google and deactivate it, to try that out too.

That's all i can contribute to your problem. Wish you good luck

Search

Search

OPNsense best practices?

H4R0

Well-Known Member

Dunuin

Distinguished Member

H4R0

Well-Known Member

Dunuin

Distinguished Member

Dunuin

Distinguished Member

Dunuin

Distinguished Member

Dunuin

Distinguished Member

H4R0

Well-Known Member

Dunuin

Distinguished Member

Dunuin

Distinguished Member

H4R0

Well-Known Member

Dunuin

Distinguished Member

H4R0

Well-Known Member

Dunuin

Distinguished Member

Ramalama

Well-Known Member

Dunuin

Distinguished Member

Ramalama

Well-Known Member