Passthrough USB device to LXC keeping the path /dev/bus/usb/00x/00y

[toxic]

Hello, I'mtrying to passthrough a USB TPU (Google Coral) to my frigate LXC.

Now the fun thing is that it sometimes it tends to change in which order it's being detected, so my config mounting /dev/bus/usb/002/002 to the lxc works fine untill for some reason the device becomes accessible under /dev/bus/usb/002/004 and then it all breaks down...

I have tried using udev to match the device ID and link the device to /dev/coral0 and like this I can sucessfully passthrough the same path /dev/coral to my LXC...

But the issue is that once in the LXC, lsusb still shows the device on bus 2 device #4 and I could not find a way to tell my frigate instance to use /dev/coral0. This is because my understanding is that the coral library will search and try to find the TPU device by itself, so I would realy need to finx a way to map /dev/bus/usb/00x/00y to the exact same x and y inside the lxc.

I have made a script to do this, it does lsusb, grabs the bus and device number and creates a symlink to /dev/coral0 on the proper /dev/bus/usb/00x/00y but seems the lib isn't following simlinks and the PU is not found by frigate...

I tried mount --bind inside the container thinking it would be like a symlink but seems it doesn-t work either...

Any idea is welcome to find a solution
I would have expected there to be a way to tell pve/lxc that device by vendor&id should be mounted in the matching /dev/bus/usb/00x/00y inside the lxc, but I can't seem to find this option...

Now I have temporarly fixed it by passing the entire bus 002 hoping it will not change often, but still, I'd rather pass only the device istead of the full bus...

Thanks in advance for any kind of help on this !

 

[dude84]

Hi, a bit late, but if someone ventures here... You can pass through an entire port into LXC instead of a dedicated USB bus/port. You will have to edit your LXC machine config file (/etc/pve/lxc/[your guest id].conf) manually, and make sure you pick the right option for create=dir, as the majority of examples will suggest you would suggest file.

That is the bit that worked for me:

...
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir

Final checks...

Proxmox host:

# lsusb
Bus 002 Device 002: ID 18d1:9302 Google Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

LXC guest:

$ ls -la /dev/bus/usb/002/
total 0
drwxr-xr-x 2 root root       80 Sep 20 13:46 .
drwxr-xr-x 3 root root       60 Sep 20 13:55 ..
crw-rw-r-- 1 root root 189, 128 Sep 20 13:46 001
crw-rw-r-- 1 root root 189, 129 Sep 20 13:46 002

I found this a lot better approach than trying to modify the config on reboot based on bus/port assigned on boot, as that one doesn't solve the issue with device id changing from 1a6e:089a Global Unichip Corp. to 18d1:9302 Google Inc.

 

[kRowone9]

Hey @dude84,

I tried your solution, but I think if the bus is shared with many other devices, Frigate fails to detect the TPU.

In my case, if I passthrough /dev/bus/usb/001/003 (Coral) everything works fine (I have then mapped in docker /dev/bus/usb/001:/dev/bus/usb/001 as stated in the docs).

Now, In my server it seems that it shares the same bus (001) for all ports (1-5):

*Nevermind the other buses (002-008) since those are from a KVM I have connected in Port1

> lsusb
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 8087:0029 Intel Corp. AX200 Bluetooth
Bus 001 Device 050: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 001 Device 003: ID 18d1:9302 Google Inc.
Bus 001 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

> lsusb -t
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/5p, 480M
    |__ Port 2: Dev 2, If 0, Class=Vendor Specific Class, Driver=cp210x, 12M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=usbfs, 480M
    |__ Port 4: Dev 50, If 0, Class=Human Interface Device, Driver=usbfs, 12M
    |__ Port 5: Dev 5, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 5: Dev 5, If 1, Class=Wireless, Driver=btusb, 12M

I tried your solution, and I see all devices in my frigate LxC properly (I wouldn't mind the container having access to all of them), but it seems frigate does not detect the TPU correctly with all the devices in that bus.

I guess I could map inside the docker container the specific device, but we'd be again with the same problem (if device ID changes, we should change the docker compose, which is even worse than changing the mapped device path in proxmox).

Any idea? I have a larger thread here with everything I've tried to solve this.

 

[toxic]

I really had no issues with passing the entire bus but I seem to have less devices connected than you. I'd try to see if passing the whole bus but having only the Coral plugged in works for you, if yes that's exactly what I have so no more luck.
I suspect it might not work for you if you weren't careful enough: the create=dir at the end is needed for passing the bus not file like passing a single device

 

[kRowone9]

I suspect it might not work for you if you weren't careful enough: the create=dir at the end is needed for passing the bus not file like passing a single device

Yes, I ensured the create=dir directive was there, and all devices were passed through and accessible within the LxC.

The issue was with frigate not identifying the device properly.
Maybe there'd be a way to indicate frigate which "vendorId" or "productId" USB to look for, to ensure it detects it properly.

I'll try again only plugging the coral, to see if that's the issue.

 

[kRowone9]

Hey @toxic

Just leaving it here for reference to anyone looking.
I was researching and this will not work in unprivileged containers.

I thought the devices where being passed through correctly, but they were being passed as nobody:nogroup instead of root:root (as in your example).

This is due to the unprivileged container not having access to root, as documented here.
In some specific situations you can change the directory owner to a user X (e.g: UID 1005) and then map the UIDs from host and guest, hence keeping the container unprivileged but having an accessible shared directory.

However for our specific usecase with bus devices, we can't change the owner of the /dev/bus/usb from root to another user without incurring in A LOT of issues, and I'm not sure if mapping root would be essentially the same as making your container privileged.

So... In a nutshell, this works for privileged containers, but not for unprivileged. Will have to look for an alternate solution, probably some type of hookscript that updates the device id.