network setup in proxmox...getting mixed results

[unassassinable]

In short, I am trying to get a pfSense VM within proxmox to be my main network router/firewall.

truths:

• I can connect to proxmox's management interface (10.1.0.10)
• I can connect to pfSense's management interface (10.1.0.1)
• I can ping vlan interfaces on pfSense (10.3.0.1) from my physical workstation (10.1.0.99)
• I cannot ping the 10.3.0.1 gateway from a virtual server located on the same network (10.3.0.10)
• pfSense has firewall on its 10.3.0.0/24 interface to allow any/any
• pfSense has a vNIC on vmbr2,tag=100
• pfSense LAN is vtnet1 (the above vNIC) with IP: 10.1.0.1
• pfSense OPT1 is vtnet1.300 with IP 10.3.0.1
• I am a Linux virtual networking noob, so I believe it is a misconfiguration on my part

Here's my switch config (Cisco Catalyst3560)

# show run interface gi 0/3
interface GigabitEhternet0/3
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 100,200,300
  switchport mode trunk
end

Server's NIC (eno3) is plugged into the above 0/3 interface.

Proxmox's /etc/network/interface (only relevant sections):

auto lo                       
iface lo inet loopback        
                              
auto eno3                     
iface eno3 inet manual        
                              
auto vmbr2                    
iface vmbr2 inet manual       
        bridge-ports eno3     
        bridge-stp off        
        bridge-fd 0           
        bridge-vlan-aware yes 
        bridge-vids 2-4094    
                              
auto vmbr2.100                
iface vmbr2.100 inet static   
        address 10.1.0.10/24  
        gateway 10.1.0.1      
#Proxmox management interface 
                              
auto vmbr2.300                
iface vmbr2.300 inet manual

I need to step away for a bit, but will respond with more information (things I've tried) when I return.

 

[unassassinable]

OK back...

I had the understanding that if you have a full trunk port on your switch, that there is minimal (if any) configuration beyond creating a vmbr on the PVE hosts. All the VLNA config would be on the Vm NICs. I have also tried the following config:

1. Trunk all the VLANs that will be part of my network on the switch except the proxmox management VLAN (100; 10.1.0.0/24) would be native VLAN. So like the following:
   

   interface GigabitEhternet0/3
     switchport trunk encapsulation dot1q
     switchport trunk native vlan 100
     switchport trunk allowed vlan 200,300
     switchport mode trunk
   end

   
   
2. Then configure an IP/gateway on vmbr2 while removing vlan (vmbr2.100) (which bridges to eno3) like so:
   

   auto eno3
   iface eno3 inet manual
   
   auto vmbr2
   iface vmbr2 inet manual
           address 10.1.0.10/24
           gateway 10.1.0.1
           bridge-ports eno3
           bridge-stp off
           bridge-fd 0
           bridge-vlan-aware yes
           bridge-vids 2-4094

   
   
3. Add a vNIC to pfSense and dont tag vlan 100, setup IP address like normal, then setup VLAN interfaces

But this caused me to completly lose access to my PVE host (had to reconfigure at the terminal).

 

[unassassinable]

after some tinkering, here is what is working....

1. Switch config:
   

   # show run interface gi 0/3
   interface GigabitEhternet0/3
     switchport trunk encapsulation dot1q
     switchport trunk allowed vlan 100,200,300
     switchport mode trunk
   end

   
   
2. Proxmox network config:
   

   auto lo
   iface lo inet loopback
   
   auto eno1
   iface eno1 inet manual
   
   auto eno2
   iface eno2 inet manual
   
   auto eno4
   iface eno4 inet static
           address 172.16.0.14/24
   #iSCSI 1
   
   auto enp4s0f0
   iface enp4s0f0 inet manual
   
   auto enp4s0f1
   iface enp4s0f1 inet static
           address 172.16.0.15/24
   #iSCSI 2
   
   auto enp5s0f0
   iface enp5s0f0 inet manual
           ovs_type OVSPort
           ovs_bridge vmbr3
   
   auto enp5s0f1
   iface enp5s0f1 inet manual
   
   auto eno3
   iface eno3 inet manual
   
   auto vmbr0
   iface vmbr0 inet static
           address 10.1.1.10/24
           bridge-ports eno1
           bridge-stp off
           bridge-fd 0
   #Proxmox corosync bridge 1
   
   auto vmbr1
   iface vmbr1 inet static
           address 10.1.2.10/24
           bridge-ports eno2
           bridge-stp off
           bridge-fd 0
   #Proxmox corosync bridge 2
   
   auto vmbr3
   iface vmbr3 inet manual
           ovs_type OVSBridge
           ovs_ports enp5s0f0
   #WAN to pfSense bridge
   
   auto vmbr2
   iface vmbr2 inet manual
           bridge-ports eno3
           bridge-stp off
           bridge-fd 0
           bridge-vlan-aware yes
           bridge-vids 2-4094
   
   auto vmbr2.100
   iface vmbr2.100 inet static
           address 10.1.0.10/24
           gateway 10.1.0.1
   #Proxmox management interface
   
   auto vmbr2.300
   iface vmbr2.300 inet static

   
   
3. pfSense has three NICs
   

This seems to be working as far as I can see. But I really do not want to have to add a new vNIC to pfSense for every VLAN we create. I would love to be able to trunk and tag at the OS. Help me Obi Wan Kenobi, you're my only hope!

 

[ce3rd]

Hi, if I understand you, you want to tag the vlans inside pfsense? Should work out of the box with vlan aware bridge - I have here a Fortigate-VM with this network config and I can asign vlans inside Fortigate (vlan 8 for internal, vlan 15 for external) - switch must have the same vlans tagged, of course

iface eno2 inet manual
link-speed 1000

auto vmbr0
iface vmbr0 inet static
address 192.168.47.11/24
gateway 192.168.47.1
bridge-ports eno2
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094

VM with single network iface

 

[unassassinable]

It should, but it just don't wanna cooperate!

One thing I noticed was, since I was wanting a trunk with the management VLAN (100) to be untagged (native vlan 100), I did not include the line in the vmbr2 "bridge-pvid 100". Then in pfSense I can put VLAN 100 on vtnet1, and all the vlans on vtnet1.###. Gonna try that on Monday. Then I can hopefully just remove all the VLANs in Proxmox (vmbr2.###).

But I was not having any success just trunking everything (without a native vlan) and configuring all the VLANs on the interface of vtnet1.### in pfSense.

 

[ce3rd]

hi, if you have native vlan100 on cisco, you don't have to care about it in PVE/pfsense - all untagged frames become vl100 when entering cisco.
I think your post #2 looks good, but there is a little mistake -> "iface vmbr2 inet manual" should be "iface vmbr2 inet static"