[SOLVED] pfsense on a pve cluster

[RobFantini]

Hello,
I am looking at this: https://pfsense-docs.readthedocs.io/en/latest/virtualization/virtualizing-pfsense-with-proxmox.html

we have a 7 node cluster.
the nodes connect to a pair of cumulus linux switches using LACP .
there is a bridge at switches with bonds and switch-ports assigned at bridge-ports

Is it possible to use two vlans in network/interfaces file instead of hardware nice for pfsense LAN and WAN? I am trying to make pfsense easy to migrate between nodes.

 

[aaron]

You could set the bridge on the PVE host to "vlan-aware". With that, guests can manage VLANs themselves. Otherwise, please post you your `/etc/network/interfaces` file so we all talk about the same stuff

 

[RobFantini]

Hello Aaron, we are using vmal-aware .
here is interfaces:

iface lo inet loopback

auto eno1
iface eno1 inet manual

auto enp5s0f1
iface enp5s0f1 inet manual

auto enp3s0f0
iface enp3s0f0 inet manual

auto enp3s0f1
iface enp3s0f1 inet manual

iface enp2s0f0 inet manual

iface enp2s0f1 inet manual

auto enp4s0f0
iface enp4s0f0 inet manual

auto enp4s0f1
iface enp4s0f1 inet manual

auto bond2
iface bond2 inet static
        address 10.11.12.10/24
        bond-slaves enp3s0f0 enp3s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond_xmit_hash_policy layer2+3
        mtu 9216
# ceph storage network

auto bond3
iface bond3 inet manual
        bond-slaves enp4s0f0 enp4s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        mtu 9216

auto vmbr3
iface vmbr3 inet static
        address 10.1.10.10/24
        gateway 10.1.10.1
        bridge-ports bond3
        bridge-stp off
        bridge-fd 0
        bridge-vids 2-500
        bridge-vlan-aware yes

auto enp2s0f0
iface enp2s0f0 inet static
    address  10.10.0.10
    netmask  255.255.255.0
# ring0

auto enp2s0f1
iface enp2s0f1 inet static
    address  10.10.1.10
    netmask  255.255.255.0
# ring1

and examples of how we set vlan id for kvm and lxc:

#LXC
arch: amd64
cores: 2
features: nesting=1
hostname: sogo
memory: 6144
net0: name=fbcadmin,bridge=vmbr3,gw=10.1.3.1,hwaddr=02:3A:DF:D8:87:1B,ip=10.1.3.143/24,tag=3,type=veth
onboot: 1
ostype: debian
rootfs: nvme-4tb:vm-105-disk-0,size=10G
swap: 4096

#KVM
bootdisk: scsi0
cores: 1
ide2: none,media=cdrom
memory: 2048
name: wireguard
net0: virtio=9E:08:F1:8A:D3:49,bridge=vmbr3,tag=3
numa: 0
onboot: 1
ostype: l26
scsi0: nvme-4tb:vm-103-disk-0,cache=writeback,discard=on,size=8G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=ed568ef7-2ffb-430c-a45a-17da35bdcd63
sockets: 2
vmgenid: a13bbd54-f27c-41db-a180-dc7c09836282

regarding "guests can manage VLANs themselves" - does that mean just set the vlan id for vm's as we do above?

also if you want i can post /etc/network/interfaces snippets from the switches.

 

[aaron]

Okay, so if I understand it correctly, you have a few VLANs in your network and you assign them to the guests by specifying the VLAN tag in the guests NIC settings. So far so good.

I probably don't yet fully grasp what you try to do. But you have internal VLAN(s) and a VLAN for WAN traffic right? The modem or upstream WAN connection is somewhere on a switch port?

Then why not either have multiple NICs assigned to the pfsense VM with the VLAN tags set? Alternatively you can give the pfsense an untagged NIC and let it configure the VLANs itself. That's what the vlan aware bridge allows you to do.

 

[RobFantini]

"Then why not either have multiple NICs assigned to the pfsense VM with the VLAN tags set?"

yes that is what we will do.
we'll set ip the wan and lan vlans to be used by pfsense at the bridge on the switch and just tag the vlans in vm config.

I thought it would work and was just wanted to verify.

thank you for the advice!

 

[bobmc]

So if you assign the WAN vmbrx and VLAN Y and the LAN vmbrw and VLAN Z, and if you have consistent VLANs and vmbr interfaces on each of the hosts as well as your physical switches, then all should work properly.

 

[RobFantini]

So if you assign the WAN vmbrx and VLAN Y and the LAN vmbrw and VLAN Z, and if you have consistent VLANs and vmbr interfaces on each of the hosts as well as your physical switches, then all should work properly.

I think just one vmbr would be needed on pve?
then vlans assigned at vm config at pve screen.

or could you give me an example of using multiple vmbrs in interface file format?

 

[RobFantini]

this is how the bond and bridge is set up at switch. note lacp lag is used to the pair of switches:

auto bond2
iface bond2
    alias bond2 on swp2
    bond-miimon 100
    bond-mode 802.3ad
    bond-slaves swp2
    bond-xmit-hash-policy layer2+3
    bridge-pvid 10
    clag-id 2
    #bridge-vids 2 3 4 7 8 10 15 20 70 131 132
    mstpctl-bpduguard yes
    mstpctl-portadminedge yes
    mtu 9216

auto bridge
iface bridge
    bridge-ports swp24 peerlink bond1 bond2 bond3 bond4 bond5 bond6 bond7 bond8 bond17 bond18 bond19 bond21 bond25 bond26 bond27 bond28 bond29 bond30 bond31 bond32
    bridge-pvid 8
    bridge-vids 2-250
    bridge-vlan-aware yes
    mstpctl-treeprio 4096

 

[RobFantini]

and pfsense plugs in here at switch. lacp is not set up yet due to a bug

auto swp24
iface swp24
 bridge-pvid 2