HIGH CPU on USB passthrough

[RaZZe]

Hi,

i have a problem with usb passthrough on KVM.
i have one KVM where i passthrough one USB device.
It is a "Ltd FT232 USB-Serial (UART) IC". This passthrough cause a high cpu usage about 30% all the time on the host. Inside this KVM, cpu is about 0% -- 2%.

when i make a strace on the Proxmox host, i see many many times

"[pid 28241] ioctl(34, USBDEVFS_REAPURBNDELAY, 0x7ffc23647978) = -1 EAGAIN (Resource temporarily unavailable)"

How i can debug this? The Device is not broken. I have tested one another device... same problem.

When you need much more information, tell me. you get it.

Mfg Daniel

 

[RaZZe]

can no one help me?

 

[t.lamprecht]

Hi, can you post the VMs config?

There would be a way with less overhead then passing through a single USB device, you could pass to the whole PCI bus where the usb port(s) is located top the VM. The advantage would be almost zero overhead, the disadvantage would be that those ports are unable to use outside from the VM.

"[pid 28241] ioctl(34, USBDEVFS_REAPURBNDELAY, 0x7ffc23647978) = -1 EAGAIN (Resource temporarily unavailable)"

How i can debug this? The Device is not broken. I have tested one another device... same problem.

A little hard to tell, could also be a problem of an USB hub.

What happens if you use the UART device directly on the host? Also for which baudrates to you configure the UART device?

 

[RaZZe]

Hi, can you post the VMs config?

There would be a way with less overhead then passing through a single USB device, you could pass to the whole PCI bus where the usb port(s) is located top the VM. The advantage would be almost zero overhead, the disadvantage would be that those ports are unable to use outside from the VM.



A little hard to tell, could also be a problem of an USB hub.

What happens if you use the UART device directly on the host? Also for which baudrates to you configure the UART device?

Hi,

This overhead with the USB Passthrough is new... i have no idea why and when. With Proxmox4 in beta state there was no problem like this.
3 Days ago, i have switched my complete Hardware and the problem is still there.


"What happens if you use the UART device directly on the host?"
There is no problem with HighCPU.

My config:

bootdisk: virtio0
cores: 1
memory: 512
net0: virtio=XX,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
smbios1: uuid=80da2b56-32db-43a6-a4a9-0543da1474e1
sockets: 1
usb0: host=2-3
virtio0: local:109/vm-109-disk-1.qcow2,size=16G

i think this is a bug. But i have no idea how to Debug this.

 

[t.lamprecht]

This overhead with the USB Passthrough is new...

You must understand that "passing through" a single USB port can not be done without an overhead, all messages from and to the USB have to be send to the VM, as a single USB port is not a separate addressable hardware unit (as seen from PCI(e)). For that you must pass through a whole usb bus (through its PCI address) as mentioned above, but I guess you do not want that ( I also have no simply to use tutorial ready).

Also for which baudrates to you configure the UART device?

??

With Proxmox4 in beta state there was no problem like this.

So can you try and older kernel, what's your current output of

pveversion -v

Also change the machine type to q35, this may help your situation:

qm set <VMID> --machine q35

 

[RaZZe]

You must understand that "passing through" a single USB port can not be done without an overhead, all messages from and to the USB have to be send to the VM, as a single USB port is not a separate addressable hardware unit (as seen from PCI(e)). For that you must pass through a whole usb bus (through its PCI address) as mentioned above, but I guess you do not want that ( I also have no simply to use tutorial ready).


??



So can you try and older kernel, what's your current output of

pveversion -v

Also change the machine type to q35, this may help your situation:

qm set <VMID> --machine q35

Passthrough a whole PCI Device is not unknown for me....
do not get me wrong, i'm not an idiot.
I understand the difference between classic USB-Passtrhough and PCI passthrough with IOMMU very well.

But this Issue with passthrough this USB Device and his HIGH CPU overhead is new! And for me it looks like a Bug.
I have this USB devices since 5 years Passthrough to KVM with Proxmox. And never i get this High CPU on the host. This is new.

Defective hardware can I exclude. Both the server and the usb device was changed, with no change on load.

Set machine to q35 don't fix this.
I have test pve and pvetest repo. no change. actually i'm on pvetest.

proxmox-ve: 4.1-48 (running kernel: 4.4.6-1-pve)
pve-manager: 4.1-33 (running version: 4.1-33/de386c1a)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-3.19.8-1-pve: 3.19.8-3
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-4.2.0-1-pve: 4.2.0-10
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-39
qemu-server: 4.0-71
pve-firmware: 1.1-8
libpve-common-perl: 4.0-59
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-50
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-13
pve-container: 1.0-61
pve-firewall: 2.0-25
pve-ha-manager: 1.0-28
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie

 

[t.lamprecht]

Passthrough a whole PCI Device is not unknown for me....
do not get me wrong, i'm not an idiot.
I understand the difference between classic USB-Passtrhough and PCI passthrough with IOMMU very well.

I only wondered as you said that an overhead with USB pass through is new to you, for the future simply ignore if I explain something already known its not meant to be offensive

But this Issue with passthrough this USB Device and his HIGH CPU overhead is new! And for me it looks like a Bug.
I have this USB devices since 5 years Passthrough to KVM with Proxmox. And never i get this High CPU on the host. This is new.

Ok now that is some information we can work with, a regression from qemu and or the KVM kernel tree could be a to blame here, I suspect qemu more. Can you recall when this started, meaning after which update?
We have to isolate the issue, there simply happens to much in each release cycle from qemu and the kernel to identify a suspecting commit from this vague description.

I would suggest installing older versions of pve-qemu (test qemu first, as the kernel needs more time (reboot)), http://download.proxmox.com/debian/dists/jessie/pve-no-subscription/binary-amd64/ search for pve-qemu, we have versions beginning from 2.4 available, install it stop and start the VM and check (use binary partitioning to fasten the testing up), if we can limit to a version where it works and one where not I can investigate more.

I'll bring a akin UART device to work tomorrow and test it if I get time, If I can reproduce it locally I can much more easier find the culprit, but I have a little less time until Friday thus no promises here yet.

Defective hardware can I exclude. Both the server and the usb device was changed, with no change on load.

If it works on the host without problems then it isn't a problem from the hardware/device but the virtualization stack.

 

[RaZZe]

I only wondered as you said that an overhead with USB pass through is new to you, for the future simply ignore if I explain something already known its not meant to be offensive



Ok now that is some information we can work with, a regression from qemu and or the KVM kernel tree could be a to blame here, I suspect qemu more. Can you recall when this started, meaning after which update?
We have to isolate the issue, there simply happens to much in each release cycle from qemu and the kernel to identify a suspecting commit from this vague description.

I would suggest installing older versions of pve-qemu (test qemu first, as the kernel needs more time (reboot)), http://download.proxmox.com/debian/dists/jessie/pve-no-subscription/binary-amd64/ search for pve-qemu, we have versions beginning from 2.4 available, install it stop and start the VM and check (use binary partitioning to fasten the testing up), if we can limit to a version where it works and one where not I can investigate more.

I'll bring a akin UART device to work tomorrow and test it if I get time, If I can reproduce it locally I can much more easier find the culprit, but I have a little less time until Friday thus no promises here yet.



If it works on the host without problems then it isn't a problem from the hardware/device but the virtualization stack.

Thanks for your reply .
I have installed "pve-qemu-kvm_2.4-9_amd64.deb" - stop and start vm.... but with no luck .


A bit overhead is normal in the Translation of USB to USB KVM . But not in the order . And unfortunately I can not understand when this began . I can only say that it was not a couple of releases .