USB passthrough issue

[Leo David]

Hello Guys,
I am really scratching my head trying to passthrough an usb device to a Windows10 vm.
The device is a "Gemalto SafeNet eToken" containing a certificate for e-signing.
The device has been detected and was appearing at "lsusb", also I have configured:

cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

- did not enabled "iommu=pt" yet

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.2.16-15-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on

After hypervisor reboot we can see:

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.014874] ACPI: DMAR 0x00000000CD3346F4 000158 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.014934] ACPI: Reserving DMAR table memory at [mem 0xcd3346f4-0xcd33484b]
[    0.826804] DMAR: IOMMU enabled
[    2.068951] DMAR: Host address width 46
[    2.068953] DMAR: DRHD base: 0x000000d5000000 flags: 0x0
[    2.068961] DMAR: dmar0: reg_base_addr d5000000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    2.068964] DMAR: DRHD base: 0x000000de900000 flags: 0x1
[    2.068970] DMAR: dmar1: reg_base_addr de900000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    2.068973] DMAR: RMRR base: 0x000000cf458000 end: 0x000000cf46ffff
[    2.068975] DMAR: RMRR base: 0x000000cf450000 end: 0x000000cf450fff
[    2.068977] DMAR: RMRR base: 0x000000cf452000 end: 0x000000cf452fff
[    2.068978] DMAR: ATSR flags: 0x0
[    2.068982] DMAR-IR: IOAPIC id 2 under DRHD base  0xd5000000 IOMMU 0
[    2.068984] DMAR-IR: IOAPIC id 0 under DRHD base  0xde900000 IOMMU 1
[    2.068986] DMAR-IR: IOAPIC id 1 under DRHD base  0xde900000 IOMMU 1
[    2.068987] DMAR-IR: HPET id 0 under DRHD base 0xde900000
[    2.068989] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    2.068990] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    2.069631] DMAR-IR: Enabled IRQ remapping in xapic mode
[    4.770440] DMAR: No SATC found
[    4.770444] DMAR: dmar0: Using Queued invalidation
[    4.770452] DMAR: dmar1: Using Queued invalidation
[    4.814048] DMAR: Intel(R) Virtualization Technology for Directed I/O
[10690.806273] DMAR: DRHD: handling fault status reg 2
[10690.806330] DMAR: [DMA Read NO_PASID] Request device [00:1a.0] fault addr 0xe8000 [fault reason 0x06] PTE Read access is not set

I have also did, although I am NOT sure was necessarily, especially regarding "vfio_pci" :

modprobe vfio
modprobe  vfio_iommu_type1
modprobe vfio_pci

Therefore we have,

lsmod | grep vfio

vfio_pci               16384  0
vfio_pci_core          94208  1 vfio_pci
vfio_iommu_type1       49152  0
vfio                   57344  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                73728  1 vfio
irqbypass              16384  80 vfio_pci_core,kvm

Then, I've added the USB device to the vm from the UI as "Use USB Vendor/Device ID" -> selected the SafeNet device -> power up the vm
Neither the above or "qm set 101 -usb0 host=0529:0620" made the token appear inside the vm.

pveversion
pve-manager/8.0.4/d258a813cfa6b390 (running kernel: 6.2.16-15-pve)

The hypervisor is a Dell PowerEdge R720 and the bios has already the "Virtualisation Technology" enabled. But SRV-IOV not.

I just dont understand where I did wrong, or anything more is needed to be configured.
Please let me know if any thoughts of misconfigurations that you may notice withing the above. Also I am not sure if I need to blacklist this device as well.
Thank you so much !

Leo

 

[leesteken]

It looks like you are confusing PCI(e) passthrough and USB (device or port) passthrough. You can try PCIe passthrough a whole USB controller, but maybe just the passthrough of a single USB port is good enough?

 

[Leo David]

Thank you @leesteken . You are right, I have to admit I do have a bit of a confusion....
However, my target is to only passthrough either a single port or the device id itself. So that this dongle to be directly presented to the windows vm.
Besides the required configuration for usb device passthrough ( that it seems that are wrong in my case) , most probably some of the above pci related settings will need to be excluded - would really appreciate if I would be pointed what those would be.
Thank you,

Leo

 

[leesteken]

However, my target is to only passthrough either a single port or the device id itself. So that this dongle to be directly presented to the windows vm.

Use USB passthrouugh ( https://pve.proxmox.com/wiki/USB_Devices_in_Virtual_Machines ) via the Proxmox web GUI > VM > Hardware > Add USB Device.
This works out of the box and does not need any changes to Proxmox like you did.
It does not always work for all USB devices or it does not always work well for low-latency or high-bandwidth devices, but please just try this first (and report any problems).

Besides the required configuration for usb device passthrough ( that it seems that are wrong in my case) , most probably some of the above pci related settings will need to be excluded - would really appreciate if I would be pointed what those would be.

You did preparations for PCIe passthrough ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough ) but they should not interfere with USB passthrough.

 

[Leo David]

Thank you for the help !
I will do a hypervisor reboot in a couple of hours, also perform an update.
Would the "intel_iommu=on" or "iommu=pt" parameters still needed in my case ?
I suspect the device is usb 1.1 - will confirm asap, once I'll check this. However, may it be a source of the issue ?
Should I try to add the usb device to the VM as "Use USB Vendor/Device ID" or preferably "Use USB Port" ?
Cheers,

Leo

 

[leesteken]

I will do a hypervisor reboot in a couple of hours, also perform an update.

You don't have to keep me up to date about everything.

Would the "intel_iommu=on" or "iommu=pt" parameters still needed in my case ?

Only if you also want to do PCIe passthrough but it is not needed for USB passthrough.

I suspect the device is usb 1.1 - will confirm asap, once I'll check this. However, may it be a source of the issue ?

It should not matter. What issue? That's not clear to me. If you do normal/basic/standard/out-of-the-box USB passthrough of the device (by device ID or port number), what problems do you have?

Should I try to add the usb device to the VM as "Use USB Vendor/Device ID" or preferably "Use USB Port" ?

Use port if you want to pass this port to the VM (regardless of what device is plugged in) or use device ID if you want to use any port (and only this device). Sometimes a devices changes it ID based on the driver that is loaded inside the VM. Maybe try port first.

 

[Leo David]

I just dont have any luck with this, something I am missing here...
I've added the token as per below screenshot:

lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 005: ID 0624:0249 Avocent Corp. Virtual Keyboard/Mouse
Bus 001 Device 004: ID 413c:a001 Dell Computer Corp. Hub
Bus 001 Device 003: ID 0529:0620 Aladdin Knowledge Systems Token JC
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

vm config:

agent: 1
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 16384
name: win10
net0: virtio=7A:3D:BA:1E:30:71,bridge=vmbr0,firewall=1
net1: virtio=A2:A9:4A:8D:AD:29,bridge=vmbr0,firewall=1,link_down=1
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=5ebf2e63-5841-4b90-9788-ee3c6c463437
sockets: 2
usb0: host=1-1.4
vga: qxl
virtio0: storage-ssd:101/vm-101-disk-0.qcow2,size=70G
vmgenid: e6e46d0c-54c2-4d09-907f-21a2568c948f

Still the usb token does not get presented to the vm, and I just dont have any clue what could be the reason, or how should I trace for the root cause.
The bios is SeaBIOS - although I am not sure if UEFI would be a requirement for this passthrough to work.
I've tried using both: "Use USB Vendor/Device ID" and "Use USB Port" options, same result.
Any thoughts ?

 

[leesteken]

Any thoughts ?

Can you boot the same VM with a recent Ubuntu Live installer ISO (just boot to the desktop and don't install it)? And run lsusb inside the VM?

 

[Leo David]

That's a great ideea - thank you !

 

[Leo David]

Oh boy...

root@ubuntu:~# lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 002: ID 0529:0620 Aladdin Knowledge Systems Token JC
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

The device is here, PVE seems to actually doing what it's supposed to do.
Really not sure how to address this issue. I may assume that since its correctly presented to ubuntu, same thing should happen on the windows one. The only difference is that on windows it just not showing up.
Wondering even if the current windows installed virtio ( guest tools) drivers would have something to do with this...

 

[leesteken]

The device is here, PVE seems to actually doing what it's supposed to do.

Indeed. Seems like it's specific to Windows.

Really not sure how to address this issue. I may assume that since its correctly presented to ubuntu, same thing should happen on the windows one. The only difference is that on windows it just not showing up.

Maybe it needs drivers on Windows? Does the Device Manager show nothing like an unknown device or something?

Wondering even if the current windows installed virtio ( guest tools) drivers would have something to do with this...

Maybe make sure to install all latest VirtIO drivers: https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.240-1/

 

[Leo David]

Oh boy, again...
Finally having some conclusions , but not a real solution yet.
1. I was running the 5.1 machine version ( due to old times used vms and templates ). Once I have upgraded the machine version to 8.1 and reinstalled latest virtio drivers, the token appeared on the vm, but with respect to point 2:
2. The token is only accessible over Spice / noVnc consoles, whereas the need is to have is available over win RDP. Reason behind, is that it's intended to be used by multiple users logged in from various places ( an applications starts for the rdp logged in users and uses the token ).

So from PVE perspective it seems it does all the things as per expected - and as always. Now I have to deal this RDP availability of the token...
I've read some posts regarding vnc to teamviewer would do the job that rdp does not.
I know that would be already out of this forum's scope, but I would appreciate if anybody have been through this situation, and have some thoughts..
Thank you so much @leesteken for the help, really appreciate !

Cheers !

 

[leesteken]

Oh boy, again...

Did I miss or overlook something?

Finally having some conclusions , but not a real solution yet.
1. I was running the 5.1 machine version ( due to old times used vms and templates ). Once I have upgraded the machine version to 8.1 and reinstalled latest virtio drivers, the token appeared on the vm, but with respect to point 2:

Interesting, I would not have guessed.

2. The token is only accessible over Spice / noVnc consoles, whereas the need is to have is available over win RDP. Reason behind, is that it's intended to be used by multiple users logged in from various places ( an applications starts for the rdp logged in users and uses the token ).

I don't see why it would only work over SPICE/noVNC. The USB passthrough, via port or device ID, should work regardless of (virtual) graphical interface. Can you share the current VM configuration (file) again?

So from PVE perspective it seems it does all the things as per expected - and as always. Now I have to deal this RDP availability of the token...
I've read some posts regarding vnc to teamviewer would do the job that rdp does not.
I know that would be already out of this forum's scope, but I would appreciate if anybody have been through this situation, and have some thoughts..

I don't understand why it the USB device would not be connected when using RDP, sorry. I really think it should be independent but I also have no recent experience with RDP.

 

[Leo David]

Hi,
Below, the vm config:

agent: 1
boot: order=virtio0
cores: 2
ide2: storage-hdd:iso/virtio-win.iso,media=cdrom,size=612812K
machine: pc-i440fx-8.1
memory: 16384
name: win10
net0: virtio=7A:3D:BA:1E:30:71,bridge=vmbr0,firewall=1
net1: virtio=A2:A9:4A:8D:AD:29,bridge=vmbr0,firewall=1,link_down=1
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=5ebf2e63-5841-4b90-9788-ee3c6c463437
sockets: 2
usb0: host=1-1.4
vga: qxl
virtio0: storage-ssd:101/vm-101-disk-0.qcow2,size=70G
vmgenid: e6e46d0c-54c2-4d09-907f-21a2568c948f

Just tested again.
VNC connection to the vm will show the token
SPICE console connection works as well.
RDP connection to the vm wil not show the token.

Found this:
https://www.entrust.com/knowledgeba...gning-certificate-with-the-remote-session-rdp
"You can't access a token that is inserted in the remote machine. This is due to the Microsoft security limitation."

And:
https://lifayk.blogspot.com/2012/07/windows-smart-card-subsystem-and-remote.html

So after all it seems to be a MS RDP protocol limitation...
I guess i'll have to sort it out somehow by using some third-party software or maybe even hardware...

https://www.donglify.net/security-keys-over-rdp/
- though, I'm not sure at the minute how safe would this be from a security perspective...

 

[furcin]

Hello everybody.

Im in the same case that Leo David.
With the same conclusion ..


Using a VM with windows.
USB passthrough = OK.

But the Token just run with SPICE or noVNC.
With RDP is impossible at time.

Dear Leo David, did you find a new way? or you are using VNC to work in your job?

regards.
David Furcin
from BR.

 

[Olaf Brenn]

Hi there, I am using a safenet dongle on the proxmox maschine but it wont show up in the list of usb devices - neither in bash using

lsusb

nore in the list of usb devices in the vm. Is there any recommendation? Thanks
Olaf
P.S. Proxmox 8.1.10

 

[Olaf Brenn]

I just dont have any luck with this, something I am missing here...
I've added the token as per below screenshot:
View attachment 63456

lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 005: ID 0624:0249 Avocent Corp. Virtual Keyboard/Mouse
Bus 001 Device 004: ID 413c:a001 Dell Computer Corp. Hub
Bus 001 Device 003: ID 0529:0620 Aladdin Knowledge Systems Token JC
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

vm config:

agent: 1
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 16384
name: win10
net0: virtio=7A:3D:BA:1E:30:71,bridge=vmbr0,firewall=1
net1: virtio=A2:A9:4A:8D:AD:29,bridge=vmbr0,firewall=1,link_down=1
numa: 0
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=5ebf2e63-5841-4b90-9788-ee3c6c463437
sockets: 2
usb0: host=1-1.4
vga: qxl
virtio0: storage-ssd:101/vm-101-disk-0.qcow2,size=70G
vmgenid: e6e46d0c-54c2-4d09-907f-21a2568c948f

Still the usb token does not get presented to the vm, and I just dont have any clue what could be the reason, or how should I trace for the root cause.
The bios is SeaBIOS - although I am not sure if UEFI would be a requirement for this passthrough to work.
I've tried using both: "Use USB Vendor/Device ID" and "Use USB Port" options, same result.
Any thoughts ?

Can I ask: What did you do that the usb token shows up. I attached mine to the Promox maschine an nothing happens.

 

[Olaf Brenn]

The token is now available on proxmox and connected to the vm but I cannot see it in the VM even when using VNC access. Is there any way to check if the dongle is available as I cannot see if it is a Vm problem or software? Thanks, Olaf

 

[Xaar37]

The token is now available on proxmox and connected to the vm but I cannot see it in the VM even when using VNC access. Is there any way to check if the dongle is available as I cannot see if it is a Vm problem or software? Thanks, Olaf

This is the way it is attended to work... You can only access this sort of tokens via the console (VNC, direct access to the console, etc...). It's not a problem with the VM or the Hypervisor.

If your aim is to automate code signing, you can try something like RemoteSignTool.