Can I give a VM control of a NIC without IOMMU?

[jerrac]

Hey All,

So, I have a really good idea, but it's looking like I might stymied until I get better hardware. Before I give up, I thought I'd ask for some help.

My idea is to make a Home Server and Firewall out of old pc parts. Specifically I'm trying this on hardware from an old Gateway tower. I've been using it as an Opnsense firewall for a few years now, but I want to do a bit more than that. Using Proxmox as host seems like a good way to open up more options.

The first part of my idea is a firewall VM (Opnsense, PFSense, ipfire, other) that controls my two Intel PCIE NICs. Then my cable modem would be plugged into one NIC, the other NIC would be for the LAN, and the NIC integrated into the motherboard would be for Proxmox to get on the LAN.

I initially thought I'd use passthrough to give the NICs to the firewall vm. So I followed the PCI Passthrough docs. I updated grub, updated /etc/modules, and add the unsafe interrupts setting.

This is my output from the commands that should tell me IOMMU isolation works.

root@vabbi:~# dmesg | grep -e DMAR -e IOMMU
[    0.096718] DMAR: IOMMU enabled
root@vabbi:~# dmesg | grep 'remapping'
root@vabbi:~# find /sys/kernel/iommu_groups/ -type l
root@vabbi:~#

The lack of output means IOMMU isolation is not working. Correct?

I'm very sure the reason is that my hardware very likely does not support it. This is a Intel® Core™ i3-2120 Processor on an Acer ipisb-vr rev 1.01 motherboard.

Before I give up, is there another way to make sure the firewall vm is the only one communicating with the world through my cable modem? Maybe something to do with setting up routes or iptable rules?

My day job is a Linux Sysadmin, but my networking skills are a bit lacking. I've been fortunate enough to have coworkers dealing with that stuff, and unfortunate enough that I haven't been able to find time to learn more. So feel free to be technical in your responses, but some help with networking stuff would be appreciated.

Thanks in advance.

 

[H4R0]

IOMMU requires vt-d support which your cpu does not have.

However there is no need to pass trough the pci cards in your case.


Just create a bridge for each nic on your hypervisor and then pass them trough as virtual nic.

e.g VM -> Hardware -> Add Network Device -> Select nic bridge

You would not notice any overhead.


You can then add a vlan tag to your vm's and create a vlan interface within your opnsense vm to isolate them.

Do not vlan tag your opnsense nic so all vlans are forwarded to it.


If you have a switch with vlan and lacp support you should go for a lacp trunk for additional redundancy.

 

[jerrac]

Ok, so on my node, System -> Network, I then create two Linux Bridge's. In on bridge's Ports/Slaves box, I put the name of one of my nics. Then in the other bridge, the other nic.

I then add those two bridges as nics to my firewall vm.

Would that cover the initial Proxmox side?

What happens when I want my Firewall to be the dhcp server? Right now Proxmox is getting it's ip address from my wireless router. Should I just configure the same static ip for it on my wireless router, and then my firewall vm?

 

[H4R0]

Yes thats right.

Generally every device can do dhcp, to let your firewall handle dhcp you must:
Disable dhcp on your router
Enable dhcp on your opnsense (under services)

Now comes the problem, do you only want your firewall to act as dhcp ?
If yes you must make sure your opnsense dhcp will give your old router ip as gateway address.
You can configure that in opnsense but does it make sense ?

I would go with dhcp and wan routing.
Usually opnsense install steps will require you to setup lan/wan right away.
I would first setup wan routing e.g connect the nic with your wireless router.
Let opnsense recieve a ip via dhcp from your router for the wan interface.
Now configure the lan interface to your needs and enable dhcp for it in opnsense.

I would go with another subnet for lan and wan.

If you now connect a device to the lan nic, it should get its ip via opnsense. You add a switch to the lan nic to get more ports.

You can also connect both nics with you wlan router but as it probably wont support vlan's it will be unsecure this way.


There are some more things to it and it might not work out on the first try, so try harder.

WAN/LAN Interface, DHCP, DNS, etc.


If you are stuck for some time e.g 30 minutes, post here and we will help you.

 

[jerrac]

Thanks.

Yes, getting the wireless router working with OPNsense is a chore....

Can't disconnect my internet right now, so I'll have to wait a bit. If I run into issues, I'll be back for help. And if things go well, I'll post a summary. Maybe it'll help others down the road.

 

[jerrac]

A couple questions have come up.

First, is there a way to manage Proxmox entirely from the host machine? As in, I'd install a GUI, and then run everything off of localhost? I have made an attempt at installing gnome-shell, but it wouldn't let me login as root, and when I created a new user, all I got was a mouse cursor and colored background when I logged in...

Second, would a managed switch allow me to assign ip addresses? As in I'd connect just my desktop and Proxmox node to it, and then that would let me configure an appropriate IP for Proxmox.

Basically, my wireless router is terrible. I assign static DHCP leases to MAC addresses, and it doesn't actually apply them. And there are other issues, like the lack of an AP mode.... So I'm trying to find ways around that it.

Thanks!

 

[H4R0]

First, is there a way to manage Proxmox entirely from the host machine? As in, I'd install a GUI, and then run everything off of localhost? I have made an attempt at installing gnome-shell, but it wouldn't let me login as root, and when I created a new user, all I got was a mouse cursor and colored background when I logged in...

In that case install debian 10 with gui of your choice and install proxmox over it. You can install your favorite browser. IMHO its much easier to use the Laptop tho.


Second, would a managed switch allow me to assign ip addresses? As in I'd connect just my desktop and Proxmox node to it, and then that would let me configure an appropriate IP for Proxmox.

If the managed switch supports a dhcp server yes, but thats a unusal setup. You only need to configure your hypervisor and firewall, once the firewall is set up you can configure dhcp mappings there.


Basically, my wireless router is terrible. I assign static DHCP leases to MAC addresses, and it doesn't actually apply them. And there are other issues, like the lack of an AP mode.... So I'm trying to find ways around that it.

You can run opnsense in combination with a cheap 40$ router that acts as access point running openwrt. Basically includes a managed 5 Port switch for free. Im using a Archer C7 V5 at home, works great.

 

[jerrac]

Heh. I have an Archer C7 v2. But it's the American version that you can't flash with third party firmware. I am not happy with that. And I'm still wondering how I had it working when I was running OPNsense on my old tower....

 

[H4R0]

Heh. I have an Archer C7 v2. But it's the American version that you can't flash with third party firmware. I am not happy with that. And I'm still wondering how I had it working when I was running OPNsense on my old tower....

The openwrt wiki doesnt say something about a american version.

I would give it a try, from the wiki you can simply flash the openwrt.bin via the gui.

It gives you full control over the AP and Switch, you can run multiple ssid's with different vlans for example.

https://openwrt.org/toh/tp-link/archer-c7-1750

 

[jerrac]

I have tried flashing many times, with several different versions of ddwrt. It doesn't work. I don't recall exactly where I found that the american version blocks third party firmware, but if I recall correctly, it had something to do with annoying FCC rules....

Oh, and it looks like you were talking about openwrt... Which is apparently different than ddwrt, well that's something to look into....