LXC container and Google Coral passthrough

[Mio3000]

Hello all,

after 2 days of googling and trying i have to use this forum...

i created LXC container from debian 10 and trying to passthrough the USB Coral device to it, buth i cant get it to work. i see the device in lsusb, but the example code gives me error:
ValueError: Failed to load delegate from libedgetpu.so.1

on proxmox host:

root@proxmox:/etc/pve/lxc# lsusb
Bus 006 Device 003: ID 18d1:9302 Google Inc.
Bus 006 Device 002: ID 04e8:6123 Samsung Electronics Co., Ltd D3 Station
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 002: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

LXC config file:

arch: amd64
cores: 4
features: mount=cifs,nesting=1
hostname: frigate
memory: 8192
nameserver: 192.168.123.252 1.1.1.1
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=F6:8C:DC:CF:83:0C,ip=dhcp,type=veth
onboot: 0
ostype: debian
rootfs: local-lvm:vm-108-disk-0,size=32G
searchdomain: 192.168.123.252
swap: 512
unprivileged: 1
lxc.cgroup.devices.allow: c 189:* rwm
lxc.cgroup.devices.allow: c 226:0 rwm
lxc.cgroup.devices.allow: c 226:128 rwm
lxc.cgroup.devices.allow: c 29:0 rwm
lxc.cgroup.devices.allow: c 120:* rwm
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: cgroup:rw
lxc.mount.entry: /dev/bus/usb/006 /dev/bus/usb/006 none bind,optional,create=dir 0,0

on the LXC:

root@frigate:~/coral/pycoral# lsusb
Bus 006 Device 003: ID 18d1:9302 Google Inc.
Bus 006 Device 002: ID 04e8:6123 Samsung Electronics Co., Ltd
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 002: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I really dont know what im doing wrong...

thank you for any help

EDIT:
on the proxmox host machine i tested the Coral, and its working correctly.
I also created an VM and added the coral to VM and also there it is working correctly, so the HW is ok, only with the LXC i have troubles

 

[marco1990]

Have you solved the Problem?

 

[Mio3000]

Have you solved the Problem?

yes, do you need help with it?

 

[marco1990]

Yes it work for me but sometimes the Proxmox did not return something. I must restart it manuell with the Button on the Intel NUC. If I did not use the Coral everthing works fine. Have you also problems like this?

 

[Mio3000]

Yes it work for me but sometimes the Proxmox did not return something. I must restart it manuell with the Button on the Intel NUC. If I did not use the Coral everthing works fine. Have you also problems like this?

no, i dont have such issues... do you have some errors in syslog?

 

[Mjhansen]

hi @Mio3000 ,

I have similar use case (Proxmox/lxc/frigate) , would you mind share how you mounted the edge_tpu usb from the container

I have this config:

root@pve01:/# lsusb | grep 18d1
Bus 002 Device 002: ID 18d1:9302 Google Inc.
root@pve01:/# ls -la /dev/bus/usb/002
total 0
drwxr-xr-x 2 root root 80 Jan 5 00:18 .
drwxr-xr-x 8 root root 160 Jan 5 00:18 ..
crw-rw-r-- 1 root root 189, 128 Jan 9 22:47 001
crw-rw-r-- 1 root root 189, 129 Jan 9 22:47 002
root@pve01:/#


root@pve01:/# more /etc/pve/lxc/117.conf
arch: amd64
cores: 4
features: nesting=1
hostname: frigate
memory: 1024
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=62:7E:FE5:90:34,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: zfsnve:basevol-114-disk-0/subvol-117-disk-0,size=20G
swap: 1024
unprivileged: 0
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: a
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0
root@pve01:/#

# Btw: tried to add these without luck:
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir 0,0
# Error in frigate
frigate.detectors.plugins.edgetpu_tfl ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.

 

[Mjhansen]

hi @Mio3000 ,

I have similar use case (Proxmox/lxc/frigate) , would you mind share how you mounted the edge_tpu usb from the container

I have this config:

root@pve01:/# lsusb | grep 18d1
Bus 002 Device 002: ID 18d1:9302 Google Inc.
root@pve01:/# ls -la /dev/bus/usb/002
total 0
drwxr-xr-x 2 root root 80 Jan 5 00:18 .
drwxr-xr-x 8 root root 160 Jan 5 00:18 ..
crw-rw-r-- 1 root root 189, 128 Jan 9 22:47 001
crw-rw-r-- 1 root root 189, 129 Jan 9 22:47 002
root@pve01:/#


root@pve01:/# more /etc/pve/lxc/117.conf
arch: amd64
cores: 4
features: nesting=1
hostname: frigate
memory: 1024
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=62:7E:FE5:90:34,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: zfsnve:basevol-114-disk-0/subvol-117-disk-0,size=20G
swap: 1024
unprivileged: 0
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: a
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0,0
root@pve01:/#

# Btw: tried to add these without luck:
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir 0,0
# Error in frigate
frigate.detectors.plugins.edgetpu_tfl ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.

EDIT: managed to get it to work, so just sharing my learnings.

The issue was most likely an obsolete usb device file in /dev/bus/usb/002 dir in lxc. Migrating frigate docker from Proxmox VM to LXC caused inference speed went down from 15 to 8ms. Did even manage to get my GPU passthrough to work with LXC, did not work with Debian VM (AMD Ryzer 7000 RENOIR integrated GPU)

So this works:
root@frigate:~# ls -la /dev/bus/usb/002/006
crw-rw---- 1 root plugdev 189, 133 Jan 10 01:27 /dev/bus/usb/002/006
# lxc 117
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/bus/usb/002 dev/bus/usb/002 none bind,optional,create=dir 0,0

## Docker-compose mapping
devices:
- /dev/dri/renderD128
- /dev/bus/usb/002/006:/dev/bus/usb/002/006
##

A bit odd observation is that the google device is lacking the desc inside frigate docker, but seems not to be the root cause.
root@frigate:~# lsusb | grep "Bus 002"
# lxc 117
Bus 002 Device 006: ID 18d1:9302 Google Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
# Docker
root@ec43b67b6044:/opt/frigate# lsusb | grep "Bus 002"
Bus 002 Device 006: ID 18d1:9302
Bus 002 Device 001: ID 1d6b:0003 Linux 5.15.83-1-pve xhci-hcd xHCI Host Controller

 

[Mio3000]

@Mjhansen - sorry for late response, do you still need my config to compare? i managed to get it working on LXC and still using it as LXC, without docker to be used.

 

[Mjhansen]

@Mjhansen - sorry for late response, do you still need my config to compare? i managed to get it working on LXC and still using it as LXC, without docker to be used.

no worries. I believe I got everything working. But do you run frigate native in LXC without docker? If so, could be interesting how this is done…

 

[Mio3000]

no worries. I believe I got everything working. But do you run frigate native in LXC without docker? If so, could be interesting how this is done…

yeah, i ment without using VM, not docker

EDIT: btw do you have experience that the LXC container where is frigate running grows slowly? in the disk space usage

 

[codemannh]

Unfortunately, I am having the same issue, but have not yet gotten it resolved. My configuration is exactly as yours is. I loaded the example classifier and it works fine in the host, but the LXC comes back with the same error the OP was getting. Tried multiple times as I read somewhere the first time didn't work and then it was run multiple times after that. Not sure what happened after that for others, but I have not reached that level of success yet.