USB passthrough to LXC problem

Assuming you restarted (reconfigured?) the NUT container correctly it appears that the NUT container cannot access the original file only the symlink, as your output would indicate:


I googled a bit on the subject & it would appear others have had similar issues with containers (though in docker) only being able to access the symlink & not the original file.

Lets try & see if a workaround helps.

What does the following show (on the PVE host):

readlink /dev/ups0
this is the docker-compose.yml:
Code:
version: '2.2'
services:
  nut-upsd:
    container_name: nut-upsd
    image: upshift/nut-upsd
    devices:
    - /dev/ups0
    environment:
    - UPS_NAME=APC
    - UPS_DRIVER=usbhid-ups
    - API_USER=master
    - API_PASSWORD=mypasswd
    network_mode: host
    restart: unless-stopped

when the usb path changes I modify only the line: - /dev/ups0
 
Replying to my own query - I found the command in the docs! Its with the --delete <string> parameter. So in your case that would be:
pct set VMID --delete dev0 (if you used dev0 originally, I'm not sure) & then reapply the new pct set .... command. Although I believe all this is not necessary - since as long as you used the same device (dev0 ?) both times, & then restarted the LXC, it will simply have been replaced.

In any case please provide, as already requested, the output for (from host):
readlink /dev/ups0

readlink -f /dev/ups0

Good luck.
the command: readlink /dev/ups0 doesn't show anything, readlink -f /dev/ups0 shows "/dev/ups0"
 
Replying to my own query - I found the command in the docs! Its with the --delete <string> parameter. So in your case that would be:
pct set VMID --delete dev0 (if you used dev0 originally, I'm not sure) & then reapply the new pct set .... command. Although I believe all this is not necessary - since as long as you used the same device (dev0 ?) both times, & then restarted the LXC, it will simply have been replaced.

In any case please provide, as already requested, the output for (from host):
readlink /dev/ups0

readlink -f /dev/ups0

Good luck.
Originally I used:
pct set 105 --dev0 path=/dev/bus/usb/001/012
and after:
pct set 105 --dev0 path=/dev/ups0
 
OK you have a rather complicated setup. You are running a docker inside an LXC.

(I must be honest, I also run a NUT server, but I have it inside the Proxmox host. Yes I know its not ideal to tamper too much with the host, but I find that with something that is totally dependent on a specific USB device, I'm not going to go down the rabbit hole of USB passthrough etc.)

But back to your problem, let's try changing the Udev rule & see if it helps. So lets try this:

Code:
nano /etc/udev/rules.d/50-apcups.rules

SUBSYSTEM=="usb", ATTRS{idVendor}=="051d", ATTRS{idProduct}=="0002", SYMLINK+="ups0", GROUP="100000", MODE="0666"

#save & exit

udevadm control --reload

udevadm trigger

Then start up LXC & see what happens.
 
now, when the USB path changes, the LXC container doesn't start, this is the error:
"TASK ERROR: Device /dev/bus/usb/001/012 does not exist"
The LXC container starts again if I type: pct set 105 --delete dev0

Maybe I have to find an easier solution, if I install NUT on proxmox host only UDEV rule is needed?
 
Hey @gfngfn256,

Problem

I've been reading this thread and running some tests, and it seems I'm stuck with this problem.
I successfully run NUT server (in its docker container) inside an LxC. I do not want to switch to running the docker directly on the host as you do.

My only issue is that everytime I reboot, a new device id is assigned to the USB, so I have to manually fix the NUT LxC passthrough.
This does not happen with my other USBs (at least not so often) such as Google Coral or a Zigbee antenna, which seem to be much more stable (in terms of device ID), but with the NUT USB its basically everytime I reboot.

Testing suggested fixes

Test 1​

I've tried to setup a UDEV symlink and directly passthrough the symlink. It does not work. The container does not recognize the device.
Also I know this is not an issue of running it within docker, since doing the same with my Zigbee LxC container also makes the antenna unrecognizable (and my zigbee2mqtt server is not within docker but directly on the LxC container).

Test 2​

I've also tried several approaches with readlinks as you suggested. In order to get full path I even tried something a bit more complex:
Bash:
pct set <vmid> --dev0 path=/dev/$(readlink /dev/coral -n | sed 's|\.\.|/dev|g')

The command works, but it basically assigns the /dev/usb/bus/XXX/XXX device, so upon reboot it fails again.

I've also tried to put the command directly within the /etc/pve/lxc/XXX.conf file, so that it "reads the link" upon starting the LxC, but it fails since you can't pass bash within that YAML config.

Test 3​

I even tried to map the device "the old fashion way" (before Proxmox supported USB passthrough) with this in the LxC config :
YAML:
lxc.cgroup.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/ups dev/bus/usb/001/001 none bind,optional,create=file

This basically maps my /dev/ups alias into a /dev/bus/usb/001/001 device inside the LxC container. But the problem persists, since /dev/bus/usb/001/001 is still unrecognized by the LxC as a device (since its a symlink).

Test 4​

I also saw this thread and tested to bind the whole bus with this.
Although it seems that with this "old fashioned way" you can map the entire bus to the LxC container regardless of it being unprivileged, it seems that frigate fails to detect the USB.
Also tried with NUT, and it does not recognize the device when there are more devices mapped to the container.

Alternative Solutions

I think that with a privileged container we could be able to share the udev symlinks between host and LxC, and make it work. But I want to avoid making my LxC's privileged.
Also, if making it privileged was an option, no udev would be necessary at all since we could pass the whole bus instead of only the device, avoiding any issues with device id changes.

Therefore the only solution I can think of at this point, is a init.d script that runs every reboot and updates the new /dev/bus/usb/XXX/XXX path with something like:
Bash:
pct set <vmid> -del dev0 && pct set <vmid> --dev0 path=/dev/bus/usb/$(lsusb| grep 051d:0002 | cut -d ':' -f1 | cut -d " " -f2,4 | sed 's| |/|g')

But not even sure if that script would successfully ran after initializing proxmox (so that pct works properly) but before containers are started.

Any other ideas?
 
Last edited:
But not even sure if that script would successfully ran after initializing proxmox (so that pct works properly) but before containers are started.

Any other ideas?
Something that may be worth looking into - along your idea - is using a hookscript (can be written in Bash) for that LXC to set that USB dev at the pre-start stage.

Hookscripts are rather thinly documented by Proxmox docs - so you will have to research this idea.
 
Something that may be worth looking into - along your idea - is using a hookscript (can be written in Bash) for that LXC to set that USB dev at the pre-start stage.

Hookscripts are rather thinly documented by Proxmox docs - so you will have to research this idea.
But if I understand correctly, hookscripts are executed within the guest, right?
If that's the case, the guest can not manage its own devices passthrough, so it wouldn't be of much use to run anything inside it once it has started (it wouldn't even start without the USB properly assigned).

Thats why I suggested a init.d script (which runs at the host level) that assigns the devices.
It would not solve "hotplug" recognition, but it would at least solve the recognition upon reboot.
However its not an ideal solution, but rather a sketchy workaround.. That's why I asked of any other ideas.

Sharing the whole bus worked properly, so Im wondering why NUT / Frigate / Zigbee2MQTT didn't recognize the device between all of them, and if there might be a way for them to identify properly which device they have to call (without hardcoding the path it of course, to ensure its always properly recognized).
 
But if I understand correctly, hookscripts are executed within the guest, right?
Wrong. They are run on the host, & you can run what you like.

At the documentation I posted (already above) - it shows you to look at the example file /usr/share/pve-docs/examples/guest-example-hookscript.pl

I'll post the output for it so you can check it here:
Code:
cat /usr/share/pve-docs/examples/guest-example-hookscript.pl

#!/usr/bin/perl


# Exmple hook script for PVE guests (hookscript config option)
# You can set this via pct/qm with
# pct set <vmid> -hookscript <volume-id>
# qm set <vmid> -hookscript <volume-id>
# where <volume-id> has to be an executable file in the snippets folder
# of any storage with directories e.g.:
# qm set 100 -hookscript local:snippets/hookscript.pl


use strict;
use warnings;


print "GUEST HOOK: " . join(' ', @ARGV). "\n";


# First argument is the vmid


my $vmid = shift;


# Second argument is the phase


my $phase = shift;


if ($phase eq 'pre-start') {


    # First phase 'pre-start' will be executed before the guest
    # is started. Exiting with a code != 0 will abort the start


    print "$vmid is starting, doing preparations.\n";


    # print "preparations failed, aborting."
    # exit(1);


} elsif ($phase eq 'post-start') {


    # Second phase 'post-start' will be executed after the guest
    # successfully started.


    print "$vmid started successfully.\n";


} elsif ($phase eq 'pre-stop') {


    # Third phase 'pre-stop' will be executed before stopping the guest
    # via the API. Will not be executed if the guest is stopped from
    # within e.g., with a 'poweroff'


    print "$vmid will be stopped.\n";


} elsif ($phase eq 'post-stop') {


    # Last phase 'post-stop' will be executed after the guest stopped.
    # This should even be executed in case the guest crashes or stopped
    # unexpectedly.


    print "$vmid stopped. Doing cleanup.\n";


} else {
    die "got unknown phase '$phase'\n";
}


exit(0);


This example one is written in Perl, but as I said above AFAIK you can also do a bash one.
 
Wrong. They are run on the host, & you can run what you like.
Oh nice, then it might indeed be a better approach than "init.d", to ensure Proxmox is up (and pct actions like set can already be invoked), but that they run before trying to start the containers.
I'll take a look at them.

Sad that Proxmox does not provide a better alternative.

I'm still wondering why one USB (front one) is properly detected as ttyUSB0 and has a device under /dev/serial/by-id/xxxx but the rest of them do not, and are only accessible via /dev/bus/usb/XXX/XXX.

If all of them were in /dev/serial/by-id, I could just passthrough these paths and forget about all this mess.
 
then it might indeed be a better approach than "init.d", to ensure Proxmox is up
The exact reasoning why I suggested it.

but that they run before trying to start the containers
You'll still have to test - with your own setup - if they are actually usable for your requirements.

Sad that Proxmox does not provide a better alternative.
Not sure what you expected?

I'm still wondering why one USB (front one) is properly detected as ttyUSB0 and has a device under /dev/serial/by-id/xxxx but the rest of them do not, and are only accessible via /dev/bus/usb/XXX/XXX.
Different buses/adapters/drivers - the land of Linux!
 
Not sure what you expected?
It would be ideal that proxmox offered a way to passthrough USB devices by vendorId/productId/serial, for example.
This way we could directly map the device from the UI to a container and be sure it will always be mapped regardless of hot plug/unplug or reboots, and without sketchy scripts that will have to be maintained, etc.

Different buses/adapters/drivers - the land of Linux!
I know.. I think it's guess the main reason is my Beelink SER8 has 2 USB controllers, one for the front port and another one for the back ports.
And the back ports controller might not be mapping properly the usbs.. but its a pitty, cause that would solve my issue.
I'm sure we could potentially fix it with some different driver / controller firmware, but don't want to dive into that... It gets messy pretty quickly

I guess its the main reason Proxmox hasn't prioritized or implemented yet the above, since most users will be able to map the /dev/serial/by-id/xxx and avoid any issues with bus/device id changing.
 
I guess its the main reason Proxmox hasn't prioritized or implemented yet the above, since most users will be able to map the /dev/serial/by-id/xxx and avoid any issues with bus/device id changing.
That & the fact that Proxmox is a HV environment where naturally the server is designed to be up 24/7, so hardly ever rebooted - and when it is its done manually by a professional IT system administrator who can deal with any of this (& more!) on the go.
 
That & the fact that Proxmox is a HV environment where naturally the server is designed to be up 24/7, so hardly ever rebooted - and when it is its done manually by a professional IT system administrator who can deal with any of this (& more!) on the go.
I've worked myself as system administrator, and if you can save time to sysadmins every time they have to do maintenance with a small and simple feature, why shouldn't you implement it.

The same reasoning could be applied to not implement many of the features it offers. The fact that usually is run by sysadmins does not suffice to implement or not a feature that could be useful.
 
@gfngfn256 Just as an update on the topic:

I've written a hookscript that bypasses the PCT lock and is able to update the container's device ID.
It works great when the "wrong device ID" exists.

The problem is that it seems that hookscripts (even pre-start stage) run when the container has started, and if the container can't start (e.g: because a USB device doesn't exist because the id is wrong), then the hookscript isn't executed at all.

So I guess the only remaining option would then be to run init.d..