[SOLVED] qemu guest as gateway connected to physical network

[stnstn70]

hello guys, I need some help with qemu configuration on proxmox

I want to run opnsense in qemu vm, the aim is to run 2-3 (opnsense in qemu and a one maybe two lxc containers) machines virtualized

the problem I've got stuck with is my ISP kinda "blocking" me because I have one MAC address of physical device (RTL8139) when the PC is being booted and then another
MAC address when the vm with bridged vmbr0 device is brought up.

What I've found so far: when I turn on the host PC my RTL8139 card sends some requests maybe those are some BOOTP requests, I don't know but I could sniff them if needed, then when the VM with OPNSence is being brought up it cannot get IP from ISP by MAC as it has different MAC address.

I tried to set net0 virtio=XX:XX:XX:XX:XX:XX to the same as physical RTL8139. Of course the idea has failed as there were messages in dmesg saying smth about source mac was the same as dest mac (as if I had duplicate mac on the net)

I alse tried to use iommu but the hardware doesn't seem to support

Maybe this is not the way it should be done? I mean the whole idea is wrong and there is some kind of best practice

here's what I've found on reddit for example:
https://www.reddit.com/r/OPNsenseFirewall/comments/a25pth/best_container_solution/eazx30x/
the guy runs multiple opnsense instances in proxmox, but of course he has another network conditions

 

[t.lamprecht]

Have you tried contacting your ISP? I mean if this a datacenter or hosted environment they really should not block such things. IMO, but some still do (and may have written this even in their TOS).

I mean you could do NAT on the host itself, but I'd guess that this defeats a bit the purpose

Your ISP probably does the same as we do when a CT firewall is set to on and MAC filter is enabled, this is not to easily avoidable.. Maybe @wbumiller has a better idea, like always for network stuff

 

[stnstn70]

Have you tried contacting your ISP? I mean if this a datacenter or hosted environment they really should not block such things. IMO, but some still do (and may have written this even in their TOS).

Oh no, this is my homelab so perhaps ISP is just protecting himself from people connecting their switches to ISP cable )))
Yes, I've contacted them right away and the answer was "you've probably requested a DHCP lease with RTL8139 MAC, we gave it to you for 86400 seconds then in 120 seconds you've requested another DHCP lease with another MAC on the same switch port but this is not allowed so you have to make DHCP release from RTL8139 MAC first"

I mean you could do NAT on the host itself, but I'd guess that this defeats a bit the purpose

thanks for your really exact understanding of the situation ))))

Your ISP probably does the same as we do when a CT firewall is set to on and MAC filter is enabled, this is not to easily avoidable.. Maybe @wbumiller has a better idea, like always for network stuff

Yes, what I'm really asking is whether I'm trying to do smth non-standard (which actually is not, I think this is a very simple and standard case) and
am I to use some "advanced" techniques like iommu or maybe some others? Or am I digging in the right direction but failing implementing it (it seems that I simply have forgotten to switch off "boot from LAN" option in bios hence I got into all these problems)?

Thanks for your help!

 

[t.lamprecht]

Yes, what I'm really asking is whether I'm trying to do smth non-standard (which actually is not, I think this is a very simple and standard case)

So you only have a single Public IP available for your network.
You IMO either have the following options:
* add a second NIC, pass that through to the opnsense, connect the WAN from your ISP there. Then add a second virtual NIC to the VM which is on the vmbr0 (default Linux bridge) from the PVE host, that's your LAN side, all devices required to access the network must be connected over that, i.e., the other physical network adapter of the PVE host (normally you'd connect a switch on that)

* The same as above, but instead of using two physical network cards/ports use VLANs to separate WAN from LAN, i.e., the VM gets one untagged (WAN) and one tagged (LAN) virtual NIC. but for that to work you need to have a switch which can handle VLANs.

 

[stnstn70]

So you only have a single Public IP available for your network.
You IMO either have the following options:
* add a second NIC, pass that through to the opnsense, connect the WAN from your ISP there. Then add a second virtual NIC to the VM which is on the vmbr0 (default Linux bridge) from the PVE host, that's your LAN side, all devices required to access the network must be connected over that, i.e., the other physical network adapter of the PVE host (normally you'd connect a switch on that)

* The same as above, but instead of using two physical network cards/ports use VLANs to separate WAN from LAN, i.e., the VM gets one untagged (WAN) and one tagged (LAN) virtual NIC. but for that to work you need to have a switch which can handle VLANs.

Thanks, understood
Yes, I have 2 NICs, I'll stick to the first variant you've proposed no problem with that so far
Couldthe WAN part be done without passthrough? Actually this was the first thing I tried (enable iommu, I took the guideline here https://scottlinux.com/2017/05/10/how-to-enable-iommu-support-in-fedora-linux/ ) but it seems that the motherboard I'm using can't handle it as I couldn't start the VM, proxmox was saying that he couldn't go without IOMMU. It's a quite outdated but speed-suitable combo of cpu/mb/mem for the abovementioned purpose.
Or must this scenario be accomplished with passthrough? Thanks for your help!

 

[t.lamprecht]

Couldthe WAN part be done without passthrough?

Yes, add a second Linux bridge without an address and without DHCP active, add the WAN NIC as bridge-port.
Then you can use that as bridge for the VM NIC which will become the WAN port of opnsense.

 

[stnstn70]

Yes, add a second Linux bridge without an address or a DHCP active, add the WAN NIC as bridge-port.
Then you can use that as bridge for the VM NIC which will become the WAN port of opnsense.

Hi, Thomas
I've managed to set the system up properly yesterday it was misconfigured as it tried to request dhcp lease from WAN vmbr1 with physical address which was my fault of course.
Everything works like a charm now, thanks for your help!

 

[t.lamprecht]

Great to hear!