[SOLVED] Strange routing - host can't connect to internet via VM gateway and PCI passthrough'd NIC.

taurolyon

Member
Sep 7, 2021
9
1
8
Sorry for the long post. I'm having to connect to my host box by KVM and there's no easy way to copy-paste from there. I had to install LXDE and a web browser to configure my firewall VM.

Anywho... My setup is on a dedicated server with a single IP address and network uplink. I am trying to setup OPNsense/PFsense as a VM with the network card passed through as the WAN, the LAN is connected to the bridge vmbr0.

I did try shifting this to vmbr1 in a later attempt so my logs below will reference vmbr1. I'm trying to use only using one bridge so I use these interchangeably.

My bad network diagram:
Untitled Diagram.drawio.png

What works:
OPNsense picked up the Intel NIC via PCI Passthrough without a problem and I was able to configure the static IPs for the LAN and WAN sides.
OPNsense can connect to the internet and communicate properly.
Guest machines can connect to vmbr0, is assigned a DHCP by OPNsense, and communicate with the internet.
Proxmox host can ping the gateway firewall IP.

What doesn't work
Proxmox host cannot communicate with the internet.

What's weird:
When VM100 is running: a tap100i0 device with a link-local IP is created.Screenshot 2023-02-11 012542.png

This tap device creates a default entry on my routing table
Screenshot 2023-02-11 012751.png
I attempted to remove the default, even tried a flush and this happens:
Screenshot 2023-02-11 014522.png
This default route remains... and any attempts to add or change the default root produces the "File exists" error.

My /etc/network/interfaces:
Screenshot 2023-02-11 013002.png
(disregard eno1, as it is passed through to VM100 and isn't present after the VM is running)
Probably bad idea: I attempted to bridge to lo, but it disregards this as the mac is all zeros (invalid).
Ping attempts from the host:
Screenshot 2023-02-11 013547.png

I'm not sure what I'm missing.
Does Proxmox allow for such a configuration where there is no physical network device on the host box?
 
So, the answer was to use ip route replace instead of add/del.

I fixed it with:
Code:
sudo ip route replace default via 172.16.0.1 dev vmbr1