Purpose
Inspired by Level1Techs, I decided to replace my old router with a single box solution that's flexible, upgradable, and efficient. That includes using Proxmox to utilize spare resources without interfering with the vital router functionality. With my current internet, my old router is a bottleneck. Good, fast routers are approaching desktop pricing and compatibility with custom software and firmware is limited. I struggled for a several days with this project, tripping on unintuitive obstacles, and following breadcrumbs of incomplete or overly technical tutorials and forum threads. I felt obligated to make a post on how I got it working. Strap in. It's not a simple process and I went heavy on the details.
Disclaimer
My Parts: Used Lenovo MiniPC M720q off ebay with i5-8400T (low power), 16GB RAM, 256GB SSD. Proprietary brackets, Mediatek MT7921 mini-PCIe Wifi 6 adapter (the Lenovo included an Intel Wifi card that can't be used for AP), and antenna off Aliexpress. 4 port 2.5GbE Realtek RTL8125BG PCIe card off Amazon. I had to put electrical tape on the bottom of my PCIe network card because the miniPC's expansion bracket left it touching some metal connectors. Thankfully, I noticed this ahead of time.
Plan your Network
The crux of this guide is the network setup. The Proxmox UI's default bridging setup will leave multiple systems exposed to your WAN ethernet, bypassing OpenWRT and potentially causing your internet to disconnect if your ISP is like mine. Instead one ethernet port will be used exclusively by OpenWRT as the WAN internet access. If OpenWRT is down, the internet is down. Proxmox's networking documentation has methods to work around this problem without exposing your network systems by using forwarding, masquerading, or VLAN tags. These methods create complexity, CPU overhead, and undermine OpenWRT's job. I am prepared to face the wrath of the virtual networking experts on this opinion. The wifi will be passed directly to OpenWRT as well.
Anyway, you have 2 choices for passing the rest of the ethernet ports to OpenWRT. A: Link all other ethernet ports into a single Proxmox virtual bridge and pass that. B: Add only the slowest port (likely the PC's original) to your default Proxmox vbridge and pass the rest directly. Either method provides full network access to all machines and VMs, but both still rely on OpenWRT for DHCP. Meaning you'd still have to set a static IP on a connected computer to access Proxmox if OpenWRT breaks. Option A ensures Proxmox "access" on all ports when OpenWRT is down, but creates some CPU overhead and might bottleneck crazy fast network cards. Directly passed ports have no overhead, but also lose Proxmox access if OpenWRT is down. Non-Option: Directly pass all ports, but have to plug in a monitor and keyboard and manually edit: 'nano /etc/network/interfaces' to re-add a port to the bridge before you can even see the webUI again if OpenWRT breaks. I went with option B.
Pre-Setup
Now we need to tinker in the "prox" "Shell". Unfortunately Proxmox's UI is lacking access to some important configuration and features, so we must resort to the command line. You may have enabled the virtualization and IOMMU in your UEFI, but it also needs enabling here. The IOMMU grouping # fix is safe for most hardware, but there is a chance of horrible network slowdown or other issues caused by bad support by the manufacturer of the device or the MB. If you're certain you don't need it, then leave out "pcie_acs_override=downstream,multifunction". the "video=..." parts help with iGPU passthrough and are also optional. Check your notes, are you running the GRUB or Systemd bootloader?
Inspired by Level1Techs, I decided to replace my old router with a single box solution that's flexible, upgradable, and efficient. That includes using Proxmox to utilize spare resources without interfering with the vital router functionality. With my current internet, my old router is a bottleneck. Good, fast routers are approaching desktop pricing and compatibility with custom software and firmware is limited. I struggled for a several days with this project, tripping on unintuitive obstacles, and following breadcrumbs of incomplete or overly technical tutorials and forum threads. I felt obligated to make a post on how I got it working. Strap in. It's not a simple process and I went heavy on the details.
Disclaimer
- I have no idea what I'm doing. This is my first time using Proxmox and the first/best justification I had as a non-homelabber...labber? Sure, non-homelabrador... for actually using VMs and Containers.
- OPN/PFsense users will find this guide helpful, but I went with OpenWRT because google tells me OPN/PFsense lacks modern integrated Wifi support, and I don't need a dedicated AP device.
- This tutorial is far from perfect. The results seem good, with few bugs, but I'm counting on the experienced people here to give me feedback so I can improve it.
- From a security/reliability standpoint, this is slightly lacking. Nothing is directly exposed to WAN except OpenWRT... which is a privileged container. If it's compromised, the whole system can be considered compromised, but at that point, your network is already having a bad day. If there's a reasonable configuration method for an unprivileged container, I'm all ears.
- While inefficient, VMs are less buggy and more secure than this container method. This mostly concerns better Wifi passthrough and the ability to run the Attended Sysupgrade package to update major OpenWRT versions. VM users may still find parts of this guide helpful.
- This is a headache and kind of a bad idea. The best/cheapest solution for the average nerd is probably still to find a deal on an actual router with good specs that's on OpenWRT's compatibility list and just do a proper "bare metal" install. I just wanted to learn something cool and maybe save some money in the future.
- The CPU/MB "must" support virtualization.
- PCIe slot(s). A 2nd slot or mini-PCIe helps avoid hunting for a compatible USB wifi dongle.
- High speed PCIe Network Card supported by OpenWRT.
- Wifi PCIe card, or mini-PCIe, or USB supported by OpenWRT. This is tricky. Cards with Intel chips do NOT work. Also, unless you pay extra or buy 2, you can NOT broadcast multiple frequencies (2.4 and 5 Ghz etc.) at the same time like with a commercial router. Forget about high speed MLO antenna arrays you get with spider-like routers.
- Optionally look for 7th gen Intel or newer and AMD Ryzen CPUs. They will have HEVC encoding support if you want a Jellyfin LXC. Saving on electricity means running a more efficient CPU model with integrated graphics. Dedicated GPUs are power hungry.
My Parts: Used Lenovo MiniPC M720q off ebay with i5-8400T (low power), 16GB RAM, 256GB SSD. Proprietary brackets, Mediatek MT7921 mini-PCIe Wifi 6 adapter (the Lenovo included an Intel Wifi card that can't be used for AP), and antenna off Aliexpress. 4 port 2.5GbE Realtek RTL8125BG PCIe card off Amazon. I had to put electrical tape on the bottom of my PCIe network card because the miniPC's expansion bracket left it touching some metal connectors. Thankfully, I noticed this ahead of time.
Plan your Network
The crux of this guide is the network setup. The Proxmox UI's default bridging setup will leave multiple systems exposed to your WAN ethernet, bypassing OpenWRT and potentially causing your internet to disconnect if your ISP is like mine. Instead one ethernet port will be used exclusively by OpenWRT as the WAN internet access. If OpenWRT is down, the internet is down. Proxmox's networking documentation has methods to work around this problem without exposing your network systems by using forwarding, masquerading, or VLAN tags. These methods create complexity, CPU overhead, and undermine OpenWRT's job. I am prepared to face the wrath of the virtual networking experts on this opinion. The wifi will be passed directly to OpenWRT as well.
Anyway, you have 2 choices for passing the rest of the ethernet ports to OpenWRT. A: Link all other ethernet ports into a single Proxmox virtual bridge and pass that. B: Add only the slowest port (likely the PC's original) to your default Proxmox vbridge and pass the rest directly. Either method provides full network access to all machines and VMs, but both still rely on OpenWRT for DHCP. Meaning you'd still have to set a static IP on a connected computer to access Proxmox if OpenWRT breaks. Option A ensures Proxmox "access" on all ports when OpenWRT is down, but creates some CPU overhead and might bottleneck crazy fast network cards. Directly passed ports have no overhead, but also lose Proxmox access if OpenWRT is down. Non-Option: Directly pass all ports, but have to plug in a monitor and keyboard and manually edit: 'nano /etc/network/interfaces' to re-add a port to the bridge before you can even see the webUI again if OpenWRT breaks. I went with option B.
Pre-Setup
- Clean up your system. Install the parts. Hook up internet, a screen, and keyboard. Make sure everything works.
- Probably update the UEFI/BIOS according to the manufacturer's driver page instructions.
- Configure your UEFI/BIOS settings. Google dedicated guides for BIOS/UEFI if you need help.
- Ensure virtualization (VT-x/AMD-V) and IOMMU (VT-d/AMD-Vi) are enabled.
- Boot on Power Failure set to "Last State".
- Disable SecureBoot and TPM garbage.
- Adjust boot order and whatever else to your preference.
- Prep a bootable USB drive to install Proxmox. I'm gonna tell you to use Ventoy for this. Having multiple dedicated USBs in restrictive FAT32 format that need reformatting for every new version is obnoxious. With Ventoy, you format once and drag as many ISOs and unattend scripts as you like into a folder.
- If possible, plug into your existing switch/router's ethernet using the port you want to designate as the "emergency" management port.
- Either by using the UEFI boot option key or by messing with boot order, boot into the USB install and select Install Proxmox VE (Graphical).
- The install target drive is fine as ext4 unless you plan on having your boot drive as part of a ZFS RAID array. Theoretically it should nuke any existing partitions on it's own. Don't pick the wrong drive.
- Country, Time, Keyboard.
- Pick your password. Proxmox insists on 8 characters. You can use the "passwd" command in the shell after this to freely change your password later.
- The email is only used for emailing yourself notifications on server events, updates, and error messages. If (like me) you don't care, then don't bother with a valid email.
- Network Configuration is the important/confusing screen.
- Management Interface: will try to default to the interface you already have plugged in. If not, you'll have to use your best judgement. Ethernet adapters start with "en".
- Hostname: They want a full web domain where the word before the first "." becomes the computer's (AKA node's) name. The advantage of a full web domain is that if you set up DNS for it, you can visit that in your browser rather than using the IP. I just went with "prox.box.internal". The this names the node "prox".
- FYI, this can be changed later by editing "/etc/hosts" and "/etc/hostname", but it becomes a problem if you have a "cluster" of nodes.
- IP Address (CIDR): 192.168.1.2 /24 was my choice. The IP address used to access this Proxmox computer. The /24 is the "CIDR" part and sets the network "size" such that only the last decimal number should be different for other computers. I'm not sure how much of a role it plays here since Proxmox won't be doing any DHCP. Easily editable later, but annoying if you get it wrong.
- Netmask: 255.255.255.0 It serves a similar role as the /24 above in defining how IP's are handled in the network.
- Gateway: 192.168.1.1 This is your router's IP. This number is the most common/default IP used by routers. I assume it is the IP of the box it's currently plugged into AND what you plan to use for your new router. Note the first 3 number sets match with the Proxmox IP.
- DNS: 192.168.1.1 Same as Gateway.
- Confirm the Summary and click install. After it completes, the computer will reboot. If you have the USB permanently set as your first boot device, you'll want to unplug it when the computer reboots or you'll boot to the installer again.
- Pay attention to your boot process after the install. You'll see a screen it pauses on for 5 seconds to let you choose OS boot options. If the screen is a blue box (most likely), that's GRUB. If it's black, that's systemd. This varies based on your computer and the filesystem you chose. Make a note which you have while you still have a monitor hooked up.
- Assuming you're plugged into an existing router with internet access and another computer, switch to using your other computer. Ender the IP Address you chose into your internet browser followed by :8006. That's "192.168.1.2:8006" if you copied my homework. That Proxmox port number is not easily changed btw. Welcome (I hope) to the Proxmox UI. Don't panic when you see the subscription nag popup. They use Winrar's business model.
- The settings that matter are found in the middle column after clicking a device from the tree on the left. The most notable menus in "Datacenter" are "Storage", "Backup", "Resource Mappings", "Directory Mappings". The Mappings menus are for passing hardware through to VMs.
- Click Resource Mappings, then click "Add" under PCI Devices (and/or USB Devices). You aren't actually adding anything, just noting your hardware device information. If any are missing or wrong, it's up to you to install drivers and do troubleshooting. You should see all your ethernet ports and wifi adapter. In my case they're listed as "RTL8125 2.5GbE Controllers" and "MT7921 802.11ax PCI Express Wireless.....etc." My Wifi adapter's bluetooth shows up under USB Devices despite not being one. Notice the IOMMU group numbers in the second column. If a device you want to pass through has the same number as one you don't, you can't pass through that device. The moment you boot the VM/CT with a bad passthrough, everything breaks. We'll fix that in a bit.
- While we're here. We can disable the email notification stuff if you like. Scroll down and click "Notifications", click "mail-to-root" in "Notification Targets" and click "Modify"... or just double click it. Disable it. Do the same for the listing in "Notification Matchers".
- Under "Datacenter" is your Node, AKA the computer itself. It'll have a green check. Mine is called "prox". Right now it just has network info and probably 2 drive partitions. local-lvm stores your VM/CT data. local stores everything else.
- Getting into how storage works is beyond this tutorial, but I will note that lvm is special in that you can give your VMs/CTs as much storage space as you like when you make them, but they will only use what they need for their existing files. This allows for over provisioning. CPU cores can also be over provisioned. Game server hosts are notorious for taking CPU over provisioning too far.
- Click "prox" or whatever your node is, then "Repositories". Unless you're a paying subscriber, you need to disable the 2 enterprise repositories. and use the "Add" button to replace them with the "No-Subscription" and "Ceph Squid No-Subscription" repos. Now you can go to "Updates" and run "Upgrade". There's a helper script out there to automate this basic setup and more configuration/cleanup, but I'm avoiding scripts in this tutorial.
Now we need to tinker in the "prox" "Shell". Unfortunately Proxmox's UI is lacking access to some important configuration and features, so we must resort to the command line. You may have enabled the virtualization and IOMMU in your UEFI, but it also needs enabling here. The IOMMU grouping # fix is safe for most hardware, but there is a chance of horrible network slowdown or other issues caused by bad support by the manufacturer of the device or the MB. If you're certain you don't need it, then leave out "pcie_acs_override=downstream,multifunction". the "video=..." parts help with iGPU passthrough and are also optional. Check your notes, are you running the GRUB or Systemd bootloader?
- A. Bootloader Commandline (GRUB)
nano /etc/default/grub- Edit a line...
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off video=vesafb:off" - Ctrl + O, Enter, Ctrl + X
update-grub
- B. Bootloader Commandline (Systemd)
nano /etc/kernel/cmdline- Paste as one line without quotes "
quiet iommu=pt pcie_acs_override=downstream,multifunction video=efifb:off video=vesafb:off" - Ctrl + O, Enter, Ctrl + X (to write Out file and eXit)
proxmox-boot-tool refresh
nano /etc/modules-load.d/vmiommu.conf(We're creating this file. Name is personal preference.)- Enter these 3 contents but 1 entry per line:
vfio vfio_iommu_type1 vfio_pci - Ctrl + O, Enter, Ctrl + X (to write Out file and eXit)
update-initramfs -u -k allapt install iw
What's iw?: This software is for managing wifi APs. It makes Proxmox "mostly" able to pass hardware functionality through the OpenWRT LXC container which will also have iw. A VM wouldn't need this step.- reboot now
- Optional: to check if loaded, run "
lsmod | grep vfio"