OpenWRT on LXC in Proxmox: how to correctly pass a USB modem?

[rhqq]

Greetings,

So I'm trying to run OpenWRT in LXC within Proxmox (reasoning at the end). I have been stopped by few issues, thus my questions. First of all, my config. It's a bag of different solutions, I just kept adding new options without removing previous ones, with hope it clicks in eventually. I'll trim it down to the necessary options once I get somewhere:

arch: amd64
cores: 2
features: mknod=1
hostname: openwrt
memory: 1024
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0E:D1:B1,type=veth
ostype: unmanaged
rootfs: local-lvm:vm-666-disk-0,size=8G
swap: 0
tty: 1
unprivileged: 1
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 65536
lxc.mount.auto: proc:mixed sys:rw cgroup:mixed
lxc.cgroup2.devices.allow: c 180:* rwm
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.mount.entry: /dev/bus/usb dev/bus/usb none bind,optional,create=dir
lxc.mount.entry: /dev/ttyUSB0 dev/ttyUSB0 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB1 dev/ttyUSB1 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB2 dev/ttyUSB2 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB3 dev/ttyUSB3 none bind,optional,create=file
lxc.mount.entry: /dev/cdc-wdm0 dev/cdc-wdm0 none bind,optional,create=file
lxc.mount.entry: /sys/devices/pci0000:00/0000:00:13.0/ sys/devices/pci0000:00/0000:00:13.0/ none bind,optional,create=dir
lxc.net.1.name: wwan0
lxc.net.1.type: phys
lxc.net.1.link: wwan0

First things first. I have to chmod `o+rw` abovementioned `/dev` entries on the Proxmox host (I'll swap that with an udev rule later on). The devices do show up within the container. I can use OpenWRTs uqmi command to talk to the ISP, but the problem starts when I want to establish a connection. Here's an excerpt from the logs of OpenWRT:

 Feb 16 17:38:12 2024 daemon.notice netifd: Interface 'T_Mobile' is setting up now
 Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (925): The interface could not be found.
 Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (1007): Stopping network T_Mobile
 Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (1007): Command failed: ubus call network.interface notify_proto { "action": 0, "link-up": false, "keep": false, "interface": "T_Mobile" } (Permission denied)
 Feb 16 17:38:12 2024 daemon.notice netifd: Interface 'T_Mobile' is now down

Thats all I have for now and I don't really know what to do next. Please advise.

Extra questions: do `iptables` and `nftables` work under LXC? How about wireguard?

FAQ:
1. Why LXC and not VM?
A: performance. I was peaking 50Mbit and 100% softirq on CPU inside of the VM, where modem and network are capable of 1Gb+. NATing and routing are also heavily impacted.
2. Why OpenWRT and not OPNSense/anyotherlinux?
A: OPNSense has no QMI/Mbim support. Previously I used ArchLinux and handcrafted `iptables`, but the config is getting more and more complex, thus I wanted a router-oriented distro. Also, main issues stay the same - performance and usb/interface passthrough


Resources I used so far:
https://lists.linuxcontainers.org/pipermail/lxc-users/2020-September/015337.html
https://forum.turris.cz/t/lxc-container-and-attach-usb-devices/4412/18
https://gist.github.com/crundberg/a77b22de856e92a7e14c81f40e7a74bd
https://forum.proxmox.com/threads/p...containers-in-proxmox-7-2.109981/#post-601753
https://gist.github.com/dragonfire1119/f3acd42414abfff762d2dad5e50f7cc6
https://forum.proxmox.com/threads/h...rsion-of-openwrt-and-run-it-on-proxmox.64786/
https://openwrt.org/docs/guide-user/virtualization/lxc#upgrading_to_the_latest_release

 

[Ralli]

This howto helped me and works:

https://coldcorner.de/2018/07/12/proxmox-usb-passthrough-fuer-lxc-container-z-wave-uzb1/

But look also at the comments, especially for the creating of udev-rules !

 

[generalproxuser]

1. Why LXC and not VM?
A: performance. I was peaking 50Mbit and 100% softirq on CPU inside of the VM, where modem and network are capable of 1Gb+. NATing and routing are also heavily impacted.

I have tried openwrt in lxc before moving to a vm. The lxc had too many usability problems. As for the vm, I would guess your vm isn't setup optimally. I have 1Gb Fiber and my openwrt vm on a zimaboard 832 with 2 cpus/1GB ram for the vm has no trouble maintaining 800Mb+ (some speed loss due to SQM) while 3 machines actively play online gaming. I prefer to use lxc whenever/wherever possible but sometimes a vm really is the most optimal environment depending on the task.

 

[rhqq]

I have tried openwrt in lxc before moving to a vm. The lxc had too many usability problems. As for the vm, I would guess your vm isn't setup optimally. I have 1Gb Fiber and my openwrt vm on a zimaboard 832 with 2 cpus/1GB ram for the vm has no trouble maintaining 800Mb+ (some speed loss due to SQM) while 3 machines actively play online gaming. I prefer to use lxc whenever/wherever possible but sometimes a vm really is the most optimal environment depending on the task.

This is impossible for USB, though. The overhead is simply MASSIVE and it is known to be a notorious culprit. There is not much to tinker with here. On a new laptop I'm easily reaching 1.7Gbit with the modem, on my Proxmox machine I'm easily at 600-700Mbit while running modemmanager directly and 50Mbit in the VM (tried Arch and OpenWRT) - I tried multiple tweaks before that.

I've also seen in this forum somewhere performance benchmarks reported OpenWRT as a container vs VM and the gains were 4-5fold in routing/nating.

 

[generalproxuser]

I personally would not use a usb modem unless it's for a mobile solution but everyone is different. The last time I played with my tmobile sim card for a data connection was with an rpi4 connected to an lte usb modem. I had to initialize it in a very specific way/order. Have you tried initializing the usb device directly on the proxmox host and see if it succeeds?

 

[rhqq]

I personally would not use a usb modem unless it's for a mobile solution but everyone is different. The last time I played with my tmobile sim card for a data connection was with an rpi4 connected to an lte usb modem. I had to initialize it in a very specific way/order. Have you tried initializing the usb device directly on the proxmox host and see if it succeeds?

yes, I've been using this method on proxmox, and previously, on archlinux and I'm using LTE, now 5G for over a decade (more like 15 years). I always had a huge success with the performance and reliability. it is the first time I've set this up on a VM and the performance is just abysmal. The modem itself is absolutely top-shelf solution for current 5G and the mmwave extension in the future, I'm getting constantly 1.3Gb+ speeds while directly connected to my laptop and peaking around 1.9Gbit at night. On the proxmox machine I was successfully seeing 600Mbit on bare metal, but I had to switch to USB2.0 port due to weird, not yet solved, issues with it. I'll investigate that once I have a reliable 300-400Mbit via LXC. The alternative is I scrub the whole thing and do it the old way - Archlinxux all the way.

 

[rhqq]

This howto helped me and works:

https://coldcorner.de/2018/07/12/proxmox-usb-passthrough-fuer-lxc-container-z-wave-uzb1/

But look also at the comments, especially for the creating of udev-rules !

Yeah, these udev rules (that I mentioned I'm going to set up, eventually) just set the `o+rw` on the desired /dev/ entries so the unprivileged container can talk to them. For testing purposes I've just set these permissions manually, since I want to avoid any permanent changes to my proxmox HV until I know what is necessary.

 

[generalproxuser]

I wish I had time soon to test this myself (for personal curiosity) since I still have a couple of lte usb modems. But I would think using a usb nic passed-through would at least emulate a similar scenario to see if usb as a wan interface in an openwrt vm would suffer the same speed loss

 

[rhqq]

I wish I had time soon to test this myself (for personal curiosity) since I still have a couple of lte usb modems. But I would think using a usb nic passed-through would at least emulate a similar scenario to see if usb as a wan interface in an openwrt vm would suffer the same speed loss

i've had exact same performance bottleneck using VM with archlinux. 100% softirq on 1 core. For whatever reason I was unable to perform any irqbalance, even when playing with it manually.

 

[rhqq]

Small update of no progress:

I've used ArchLinux LXC container using the stock template. I've updated the configuration to reflect the OpenWRT one I've attached in the first post. I can see the devices, I can talk with them using qmicli. ModemManager required an extra override to start (because by default it does not start in containers) - but upon starting, it reports no modems found. Something is clearly missing, but I have no idea what, or where to look for it.

 

[generalproxuser]

Looking at my notes from one of my "one off" proxmox setups, the only items in my conf file for my lxcs and a usb device is the /dev/devicename, /dev/bus/usb and the lxc cgroup device allow lines. I am using udev rules to set the perms and make the device name static but that was after getting the usb device working manually. Since you have a bunch of entries in your conf for the usb device maybe try going back to step 0 with identifying the usb device (/dev/devname) and its cgroup and then try passing the /dev/bus/usb/(devname associated) directory. On my usb lxc, owner/group is set by udev rule to the sub uid/gid and mode is set to 0664.

 

[rhqq]

Looking at my notes from one of my "one off" proxmox setups, the only items in my conf file for my lxcs and a usb device is the /dev/devicename, /dev/bus/usb and the lxc cgroup device allow lines. I am using udev rules to set the perms and make the device name static but that was after getting the usb device working manually. Since you have a bunch of entries in your conf for the usb device maybe try going back to step 0 with identifying the usb device (/dev/devname) and its cgroup and then try passing the /dev/bus/usb/(devname associated) directory. On my usb lxc, owner/group is set by udev rule to the sub uid/gid and mode is set to 0664.

I had assumed that passing the whole `/dev/bus/usb` for testing will do a trick, but sure, I'll make a test with a precision-cut path. My gut feeling is I'm missing something network-related, because I can talk to all the devs manually no problem.

 

[ohne]

Did you have any success? I had/have the same problem with an Wifi card, but using:

lxc.net.1.name: wlan0
lxc.net.1.type: phys
lxc.net.1.link: wlan0

helped to bring the interface into OpenWrt LXC, but there is still a performance loss and it doesn't feel like the "real solution". Speeds are around 250mbit in my case. When I open the AP directly on proxmox via hostapd I get like 400-500.

I found another "dirty" solution for my wifi problem, but I don't know if it is possible with wwan: I was able to bridge my wlan0 interface with an vmbr device. Proxmox doesn't allow it, but it is possible in debian. I created an empty bridge in Proxmox, let's say vmbr99. In your proxmox host wwan0 network interface settings you add an "ifup brctl addif vmbr99 wwan0" command. Like this, when your interface comes up, it is added to the bridge and that bridge can act as a normal ethernet interface to you openwrt VM and gets full speed. As I said: I don't know if it is possible with an wwan0 device in regards of iptables etc., but for wlan0 it is working. The vmbr99 is acting like a normal interface and since it has no static IP, hosts which connect to the AP get their IP from the OpenWrt DHCP server.

In my opinion in this setup when you configure the vmbr99 as wan interface in OpenWrt you should be able to receive the IP and also create firewall rules. Your proxmox host would act as a bridge device... But well, in theory

 

[rhqq]

Sorry, I haven't had time to focus on this topic recently. it is on the back burner for me, as the next objective is to actually analyse shell scripts responsible to bring up my modem connection. That'll let me find missing components of the puzzle.