[TUTORIAL] OpenWRT vm configuration for homelab/gaming router

[ladymilch]

Got OpenWRT spooled up and configured on Proxmox! I have it on my old amd fx8350 with 64G of ram, a ZFS pool, and a pcie i340-T4 NIC for future use and more learning.
The first two links are what you'll need to guide you, plus hopefully both the other links and this post help. Will keep adding as I discover and test, so check back.

install openwrt in prox

gets the net working


serial console for guest>vnc. copy paste etc.

The first two ports of the i340-t4 are made into 3 virtual bridges for wan and lan and a third for my integrated nic too. in prox, had to add the created openwrt disk to the boot menu under options in the vm and uncheck 'net', and also had to add dns server to the hosts in the gui to bootstrap the pkg updaters to the net. In the openwrt vm install 'apt update && apt install luci luci-app-sqm'. That gets you to the web gui and cake instead of it refusing to connect. ping 8.8.8.8 and google for testing.

Couldn't get nic PCIe passthrough working with my setup, spent quite a bit of effort. Doesn't appear to be needed for nic.
virtio-scsi is ~30% slower than the zfs running native on proxmox, but is still fast af. 1.4 GB/s vs 1.0GB/s writing /dev/zero to it with dd, less pronounced with more realistic workloads.

In the hardware menu for the openwrt vm:
Q35>i440FX
add a serial port for copy paste. use xterm.js as the console
I gave openwrt 4 cores with the host cpu and a 3.5 virtual core limit.
my zfs is for lan/webcache etc so has no redundancy, so all vms are hosted on lvm/thin depending on their nature, scheduled snapshots/backups to two separate disks. no cloud backup yet.

Mostly the goal is using cake/sqm to keep wan latency in check (I play a lot of league of legends and latency/jitter really mess with it if the wan link approaches saturation) and as a homelab/server (games AD/LDAP IoT etc) just making a new bridged vlan for each of the four ports and passing them to openwrt, one of the ports goes to a router in ap mode. would love to hear of how to do this better, the layout and also firewalls/security/IPS


Figured I'd start a post to both get help and leave a record for future people looking to do something similar. Interested in exploring other options as to layout etc. too. Compiling links to other sources and guides pertaining to this post overtime aswell, check back or feel free to add, seen a few deadends people have had I'm sure we can figure out.

#/etc/network/interfaces on proxmox auto lo iface lo inet loopback iface enp6s0 inet manual iface enp1s0f0 inet manual iface enp1s0f1 inet manual iface enp1s0f2 inet manual iface enp1s0f3 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.1.2/24 gateway 192.168.1.1 bridge-ports enp6s0 bridge-stp off bridge-fd 0 auto vmbr1 iface vmbr1 inet manual bridge-ports enp1s0f0 bridge-stp off bridge-fd 0 auto vmbr2 iface vmbr2 inet static address 192.168.1.30/24 bridge-ports enp1s0f1 bridge-stp off bridge-fd 0

#/etc/config/network on openwrt config interface 'loopback' option ifname 'lo' option proto 'static' option ipaddr '127.0.0.1' option netmask '255.0.0.0' config interface 'lan' option type 'bridge' option ifname 'eth0' option proto 'static' option ipaddr '192.168.1.1' option netmask '255.255.255.0' option ip6assign '60' config interface 'wan' option ifname 'eth1' option proto 'dhcp' option peerdns '0' config interface 'wan6' option ifname 'eth1' option proto 'dhcpv6'


wanna do a steam/lan/web cache/10gbe+ nas among other things
also interested in making zfs go brrrrr
3 openwrt VMs in a cluster could be fun.
pppoe server
link teaming
Performance>redundancy. recovery will take little time vs what time speed can save.

amd_iommu=on and AMD-V/IOMMU is enabled in bios etc. pcie passthrough isn't absolutely needed but eventually I'm looking to put my 3x rx580 mining gpus on a VM aswell. fx8350 doesnt support AMD-Vi but shouldn't be completely necessary from my understanding. Probably have to do something like this guy did without a passthrough, lmk.
https://forum.proxmox.com/threads/how-to-add-second-nic.40905/#post-207414

https://pve.proxmox.com/wiki/Pci_passthrough#Enable_the_IOMMU

'/etc/modules’
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

root@ddve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.106-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt

root@ddve:~#
[ 1.274324] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 13.581302] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[ 13.581303] AMD-Vi: AMD IOMMUv2 functionality not available on this system

(last two lines I believe are just warnings about not having AMD-Vi)

 

[black.jmyntrn]

I registered just to say you da man and that I am on the verge of this today and this is the best post on the internet about this. I've gone through numerous gLinet routers trying to load balance 4 cellular hot spot devices with MWAN3 and a VM on a nuc appears to be the best solution in my Overland Toyota Prius.
I'd love to have some sort of carrier aggregation with multiple USB/WIFI WAN devices, the thought of 2-3 OPENWRT instances running for each WAN or something like that is floating in my head, would need to get past 1 OPENWRT first.

will let you know how I end out on this, thanks again!

 

[ladymilch]

I registered just to say you da man and that I am on the verge of this today and this is the best post on the internet about this. I've gone through numerous gLinet routers trying to load balance 4 cellular hot spot devices with MWAN3 and a VM on a nuc appears to be the best solution in my Overland Toyota Prius.
I'd love to have some sort of carrier aggregation with multiple USB/WIFI WAN devices, the thought of 2-3 OPENWRT instances running for each WAN or something like that is floating in my head, would need to get past 1 OPENWRT first.

will let you know how I end out on this, thanks again!

Glad to help, literally why I made this, not a lot of good info on this. Also, I do link agg for my dsl with a mikrotik routerboard fwiw, not virtualized but it could be done virtually with routerOS and an x86 license. Cloud hosted router should 'just-work'

 

[black.jmyntrn]

Glad to help, literally why I made this, not a lot of good info on this. Also, I do link agg for my dsl with a mikrotik routerboard fwiw, not virtualized but it could be done virtually with routerOS and an x86 license. Cloud hosted router should 'just-work'

which model routerboard? not so much concerned with it being virtualized, just want to be able to effectively manage 4+ WAN connections ranging from USB LTE Modem, WAN port on the router, and WIFI interfaces, I have multiple cell carriers on different devices and for a while, all worked good. lately, all of the changes with 5g, antennas, and the US ban on new China chipsets have had me juggling all the carriers along with their changes plus gl.inet simply not producing a travel router for extremists like myself. just need to get by until I put together a unit with a board that has 4-6 m.2 slots each having its own modem with its own carrier, can't find the one I saw originally but it was nice... just not in the budget now.

it appears ill have to build my own "router" and have it be a powerful nuc or other.

 

[ladymilch]

If you can find a low power board that support pcie bifurcation and has enough pcie lanes your can try a 4 port nvme only pcie adapter. USB NICs might be easier even, just pass them all through to your router vm and bridge back out