Single server tutorial
Two separate VMs with FreeBSD 14.2 RC installed and running.
I wanted to avoid VLAN for family LAN, which relies on DHCP. Therefore a bit extra hardware is needed.
Key lessons learned:
Virtual Nic alias defined in rc.conf should be a /24 for lan to work, not /32 as some guides circulating on if you want LAN to work with DHCP.
Numbering of aliases in virtual Nic defined in rc.conf needs to start with number zero (example: ifconfig_re1_alias0)
vid and password defined in rc.conf can be different for WAN vs LAN
For unbound.conf it works well to use "interface-automatic: yes" when using CARP
Hardware
A Single hardware server/computer for both VMs. (not the typical dual hardware server used in this example.)
IO Crest 6x 2.5GbE PCIe NIC, where I pass through 3 of the 6 ports go to BSD1 VM server, and the residual 3 of 6 ports are passed through to second server BSD2 VM server.
3 switches: 2 small switches (Called A & B) next to Proxmox Server, and one additional regular size switch where the children LAN computers all connect (called C).
Switch A will be for WAN redundancy ( connects to Internet)
Switch B will be for LAN redundancy ( connects to Switch C)
CARP - to install add to each file:
cat /boot/loader.conf ( same on both VMs )
carp_load="YES"
cat /etc/sysctl.conf (same on both VMs )
net.inet.ip.forwarding=1
net.inet.carp.allow=1
############# net.inet.carp.preempt=1. ### optional, start without
cat /etc/rc.conf (MASTER rc.conf on BSD1 VM)
gateway_enable="YES"
defaultrouter="192.168.2.100"
ifconfig_re0="inet 192.168.2.110/24"
ifconfig_re0_alias0="vhid 1 advskew 2 pass acfe4bbd1d936b7b985b2348491f159c alias 192.168.2.140/24"
ifconfig_vtnet1="192.168.3.3/24"
pfsync_enable="YES"
pfsync_syncdev="vtnet1"
ifconfig_re1="10.10.10.3/24"
ifconfig_re1_alias0="vhid 2 advskew 2 pass 1661743630351da9c3b937aa3d3b1feb alias 10.10.10.1/24"
dhcpd_enable="YES"
cat /etc/rc.conf (BACKUP rc.conf on BSD2 VM)
gateway_enable="YES"
defaultrouter="192.168.2.100"
ifconfig_re0="inet 192.168.2.120/24"
ifconfig_re0_alias0="vhid 1 advskew 2 pass acfe4bbd1d936b7b985b2348491f159c alias 192.168.2.140/24"
ifconfig_vtnet1="192.168.3.2/24"
pfsync_enable="YES"
pfsync_syncdev="vtnet1"
ifconfig_re1="10.10.10.2/24"
ifconfig_re1_alias0="vhid 2 advskew 2 pass 1661743630351da9c3b937aa3d3b1feb alias 10.10.10.1/24"
dhcpd_enable="YES"
cat /usr/local/etc/dhcpd.conf (same on both VMs)
subnet 10.10.10.0 netmask 255.255.255.0 {
range 10.10.10.1 10.10.10.254;
option routers 10.10.10.1;
option domain-name-servers 10.10.10.1;
}
cat /etc/unbound/unbound.conf (same on both VMs)
server:
interface: 127.0.0.1
interface-automatic: yes
access-control: 192.168.2.0/24 allow
access-control: 10.10.10.0/24 allow
private-address: 10.10.0.0/16
cat /etc/pf.conf (same on both VMs)
ext_if = "re0"
re1_lan = "re1"
PFSync = "vtnet1"
nat on $ext_if from 10.10.10.1/24 to !10.10.10.1 -> 192.168.2.140
pass quick on $ext_if proto carp keep state (no-sync)
pass quick on $re1_lan proto carp keep state (no-sync)
pass quick on $PFSync proto pfsync keep state (no-sync)
# LAN Setup re1
pass in on $re1_lan
# block return in quick on $re1_lan proto { udp tcp } to ! $re1_lan port { 53 853 }. #### For some reason this BLOCK command does not work with CARP on $re1_lan, and needed to be commented out, to be further tested
pass out on $re1_lan inet keep state
Credit to Calomel.org that has a good tutorial on CARP, but not in conjunction with Unbound and Proxmox, hence this added guide.
https://calomel.org/pf_carp.html
Two separate VMs with FreeBSD 14.2 RC installed and running.
I wanted to avoid VLAN for family LAN, which relies on DHCP. Therefore a bit extra hardware is needed.
Key lessons learned:
Virtual Nic alias defined in rc.conf should be a /24 for lan to work, not /32 as some guides circulating on if you want LAN to work with DHCP.
Numbering of aliases in virtual Nic defined in rc.conf needs to start with number zero (example: ifconfig_re1_alias0)
vid and password defined in rc.conf can be different for WAN vs LAN
For unbound.conf it works well to use "interface-automatic: yes" when using CARP
Hardware
A Single hardware server/computer for both VMs. (not the typical dual hardware server used in this example.)
IO Crest 6x 2.5GbE PCIe NIC, where I pass through 3 of the 6 ports go to BSD1 VM server, and the residual 3 of 6 ports are passed through to second server BSD2 VM server.
3 switches: 2 small switches (Called A & B) next to Proxmox Server, and one additional regular size switch where the children LAN computers all connect (called C).
Switch A will be for WAN redundancy ( connects to Internet)
Switch B will be for LAN redundancy ( connects to Switch C)
CARP - to install add to each file:
cat /boot/loader.conf ( same on both VMs )
carp_load="YES"
cat /etc/sysctl.conf (same on both VMs )
net.inet.ip.forwarding=1
net.inet.carp.allow=1
############# net.inet.carp.preempt=1. ### optional, start without
cat /etc/rc.conf (MASTER rc.conf on BSD1 VM)
gateway_enable="YES"
defaultrouter="192.168.2.100"
ifconfig_re0="inet 192.168.2.110/24"
ifconfig_re0_alias0="vhid 1 advskew 2 pass acfe4bbd1d936b7b985b2348491f159c alias 192.168.2.140/24"
ifconfig_vtnet1="192.168.3.3/24"
pfsync_enable="YES"
pfsync_syncdev="vtnet1"
ifconfig_re1="10.10.10.3/24"
ifconfig_re1_alias0="vhid 2 advskew 2 pass 1661743630351da9c3b937aa3d3b1feb alias 10.10.10.1/24"
dhcpd_enable="YES"
cat /etc/rc.conf (BACKUP rc.conf on BSD2 VM)
gateway_enable="YES"
defaultrouter="192.168.2.100"
ifconfig_re0="inet 192.168.2.120/24"
ifconfig_re0_alias0="vhid 1 advskew 2 pass acfe4bbd1d936b7b985b2348491f159c alias 192.168.2.140/24"
ifconfig_vtnet1="192.168.3.2/24"
pfsync_enable="YES"
pfsync_syncdev="vtnet1"
ifconfig_re1="10.10.10.2/24"
ifconfig_re1_alias0="vhid 2 advskew 2 pass 1661743630351da9c3b937aa3d3b1feb alias 10.10.10.1/24"
dhcpd_enable="YES"
cat /usr/local/etc/dhcpd.conf (same on both VMs)
subnet 10.10.10.0 netmask 255.255.255.0 {
range 10.10.10.1 10.10.10.254;
option routers 10.10.10.1;
option domain-name-servers 10.10.10.1;
}
cat /etc/unbound/unbound.conf (same on both VMs)
server:
interface: 127.0.0.1
interface-automatic: yes
access-control: 192.168.2.0/24 allow
access-control: 10.10.10.0/24 allow
private-address: 10.10.0.0/16
cat /etc/pf.conf (same on both VMs)
ext_if = "re0"
re1_lan = "re1"
PFSync = "vtnet1"
nat on $ext_if from 10.10.10.1/24 to !10.10.10.1 -> 192.168.2.140
pass quick on $ext_if proto carp keep state (no-sync)
pass quick on $re1_lan proto carp keep state (no-sync)
pass quick on $PFSync proto pfsync keep state (no-sync)
# LAN Setup re1
pass in on $re1_lan
# block return in quick on $re1_lan proto { udp tcp } to ! $re1_lan port { 53 853 }. #### For some reason this BLOCK command does not work with CARP on $re1_lan, and needed to be commented out, to be further tested
pass out on $re1_lan inet keep state
Credit to Calomel.org that has a good tutorial on CARP, but not in conjunction with Unbound and Proxmox, hence this added guide.
https://calomel.org/pf_carp.html
