Setting up a Proxmox VM to run PfSense as main router that will initiate PPPOE session with the ISP.

[tlex]

I'm facing a strange issue where sometimes the vm wont be able to negociate PPPOE session with the ISP.
Setup :
- the setup is using a fiber connection enp4s0f1 bridged to vmbr1 (thats the only nic on that bridge and that bridge is presented only to the Pfsense vm)
- I have to go with the bridge option as I can't passthrough the nic card directly (all the IOMMU are not in the same group)
- The ISP is running on Vlan35
- Proxmox Firewall is disabled on the vm
- I've setup vmbr1 not to be Vlan aware ( I also tried with Vlan aware on)
- Vmbr1 and Enp4s0f1 has no IP assigned
- The VM nic (VirtIO) is tagged for Vlan35

For some reasons that I totally don't understand, Pfsense was not be able to initiate PPPOE session until I shutdown the VM and changed the vm NIC mac address to something totally different from what it was givin when created.
Could that pppoe issue be linked with the promiscuous mode on Vmbr1 ? or bridge_ageing ?
Anyone with a similar experience or advice ?

/etc/network/interfaces:
auto lo iface lo inet loopback auto enp6s0 iface enp6s0 inet manual auto enp4s0f0 iface enp4s0f0 inet manual auto enp4s0f1 iface enp4s0f1 inet manual auto bond0 iface bond0 inet manual bond-slaves enp4s0f0 enp6s0 bond-miimon 100 bond-mode active-backup bond-primary enp4s0f0 auto vmbr0 iface vmbr0 inet static address 10.32.50.4/24 gateway 10.32.50.1 bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-please off #LAN auto vmbr1 iface vmbr1 inet manual bridge-ports enp4s0f1 bridge-stp off bridge-fd 0 #WAN

ip -details link:
4: enp4s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000 link/ether 00:0a:f7:50:a0:82 brd ff:ff:ff:ff:ff:ff promiscuity 2 minmtu 46 maxmtu 9600 bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.0:a:f7:50:a0:82 designated_root 8000.0:a:f7:50:a0:82 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 46 numrxqueues 16 gso_max_size 65536 gso_max_segs 65535 portid 000af750a082 7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 00:0a:f7:50:a0:82 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.0:a:f7:50:a0:82 designated_root 8000.0:a:f7:50:a0:82 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 163.02 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 9: vmbr1v35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 42:53:47:03:a4:2a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.42:53:47:3:a4:2a designated_root 8000.42:53:47:3:a4:2a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 130.21 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 10: enp4s0f1.35@enp4s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1v35 state UP mode DEFAULT group default qlen 1000 link/ether 00:0a:f7:50:a0:82 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 0 maxmtu 65535 vlan protocol 802.1Q id 35 <REORDER_HDR> bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.42:53:47:3:a4:2a designated_root 8000.42:53:47:3:a4:2a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

Pfsense vm config:
bios: ovmf boot: order=virtio0;net0 cores: 8 memory: 8192 meta: creation-qemu=6.1.1,ctime=1645039349 name: PfSense net0: virtio=00:0a:f7:50:a0:83,bridge=vmbr1,tag=35 net1: virtio=06:5A:35:03:C6:5B,bridge=vmbr0 numa: 0 onboot: 1 ostype: l24 protection: 1 scsihw: virtio-scsi-pci serial0: socket smbios1: uuid=76400d77-d743-4f33-ab04-28ce65a96752 sockets: 2 startup: order=1,up=60 vga: std virtio0: R1_1.6TB_SSD_EVO860:vm-1001-disk-0,size=32G vmgenid: f058a37c-f729-40e3-8577-06b54fc60b31

pfsense config for pppoe (nothing fancy here !):

pfsense : <wan> <if>pppoe0</if> <blockpriv></blockpriv> <blockbogons></blockbogons> <descr><![CDATA[WAN]]></descr> <spoofmac></spoofmac> <enable></enable> <ipaddr>pppoe</ipaddr> </wan>

pppoe error message in loop :
Time Process PID Message Feb 16 19:08:06 ppp 13896 [wan_link0] Link: reconnection attempt 6 in 4 seconds Feb 16 19:08:06 ppp 13896 [wan_link0] LCP: Down event Feb 16 19:08:06 ppp 13896 [wan_link0] Link: DOWN event

 

[gurubert]

- I've setup vmbr1 not to be Vlan aware ( I also tried with Vlan aware on)
- The VM nic (VirtIO) is tagged for Vlan35

If the bridge is not VLAN aware you cannot assign a VLAN ID in the VM NIC configuration.
You need to create a VLAN interface (enp4s0f1.35) on the Proxmox host and use that as port for the bridge vmbr1.