Increase maximum USB devices in vm.conf

[apax]

I am following up from the suggestion below. I managed to have 8 USB disk passthrough to one virtual machine. 4 of them are USB3 and I would imagine that this configuation would work for 8 USB3 disks.

qm set 201 -args "-device nec-usb-xhci,id=xhci1,addr=0x1b,bus=pci.1 \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=4,id=dsk5Tb \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=3.1.1,id=disk4Tb \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=3.1.2,id=diska3Tb \
-device nec-usb-xhci,id=xhci2,addr=0x1c \
-device usb-host,bus=xhci2.0,hostbus=3,hostport=3.1.3,id=diskb3Tb \
-device usb-host,bus=xhci2.0,hostbus=2,hostport=3.4,id=disk2Tb \
-device usb-host,bus=xhci2.0,hostbus=2,hostport=3.2,id=diska6Tb \
-device nec-usb-xhci,id=xhci3,addr=0x1d \
-device usb-host,bus=xhci3.0,hostbus=2,hostport=3.3,id=diskb6Tb \
-device usb-host,bus=xhci3.0,hostbus=2,hostport=3.1.4,id=diskc6Tb"

with this configuration I have:
qm> info usb
Device 0.2, Port 1, Speed 12 Mb/s, Product QEMU USB Tablet, ID: tablet
Device 1.1, Port 1, Speed 5000 Mb/s, Product Elements 25A3, ID: dsk5Tb
Device 1.2, Port 2, Speed 5000 Mb/s, Product Elements 107C, ID: disk4Tb
Device 1.3, Port 3, Speed 5000 Mb/s, Product Elements 107C, ID: diska3Tb
Device 2.1, Port 1, Speed 5000 Mb/s, Product Elements 107C, ID: diskb3Tb
Device 2.2, Port 2, Speed 480 Mb/s, Product Ext HDD 1021, ID: disk2Tb
Device 2.3, Port 3, Speed 480 Mb/s, Product JM20336 SATA, USB Combo, ID: Diska1Tb
Device 3.1, Port 1, Speed 480 Mb/s, Product Ext HDD 1021, ID: Diskb1Tb
Device 3.2, Port 2, Speed 480 Mb/s, Product JM20336 SATA, USB Combo, ID: Diskc1Tb

On the other hand, if I configure in this way:
qm set 201 -args "-device nec-usb-xhci,id=xhci,addr=0x1b,bus=pci.1 \
-device usb-host,bus=xhci.0,hostbus=3,hostport=4,id=dsk5Tb \
-device usb-host,bus=xhci.0,hostbus=3,hostport=3.1.1,id=disk4Tb \
-device usb-host,bus=xhci.0,hostbus=3,hostport=3.1.2,id=diska3Tb \
-device usb-host,bus=xhci.0,hostbus=3,hostport=3.1.3,id=diskb3Tb \
-device usb-host,bus=xhci.0,hostbus=2,hostport=3.4,id=disk2Tb \
-device usb-host,bus=xhci.0,hostbus=2,hostport=3.2,id=diska1Tb \
-device usb-host,bus=xhci.0,hostbus=2,hostport=3.3,id=diskb1Tb \
-device usb-host,bus=xhci.0,hostbus=2,hostport=3.1.4,id=diskc1Tb"

I have 5 of these diskc that drops down to USB1 performance:
qm> info usb
Device 0.2, Port 1, Speed 12 Mb/s, Product QEMU USB Tablet, ID: tablet
Device 1.1, Port 1, Speed 5000 Mb/s, Product Elements 25A3, ID: dsk5Tb
Device 1.3, Port 2, Speed 5000 Mb/s, Product Elements 107C, ID: disk4Tb
Device 1.5, Port 3, Speed 5000 Mb/s, Product Elements 107C, ID: diska3Tb
Device 1.2, Port 4, Speed 12 Mb/s, Product QEMU USB Hub
Device 1.4, Port 4.1, Speed 12 Mb/s, Product Elements 107C, ID: diskb3Tb
Device 1.6, Port 4.2, Speed 12 Mb/s, Product Ext HDD 1021, ID: disk2Tb
Device 1.7, Port 4.3, Speed 12 Mb/s, Product JM20336 SATA, USB Combo, ID: diska1Tb
Device 1.8, Port 4.4, Speed 12 Mb/s, Product Ext HDD 1021, ID: diskb1Tb
Device 1.9, Port 4.5, Speed 12 Mb/s, Product JM20336 SATA, USB Combo, ID: diskc1Tb

So the solution which works for me is to create multiple xhci devices.

If anybody sees a problem with this config, let me know....

What we ended up doing was to use the qm "-args" command to add our USB devices, which allows you to specify more devices than the 5 the GUI/conf permits. For example:

qm set 100 -args "-device usb-ehci,id=ehci,addr=0x5 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.1 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.3 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.4 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.5 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.7 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.8 \"

However, this has an untended consequence like that I mentioned earlier in that once you add the 6th device, it creates a nested virtual hub behind the first one. While this shouldn't necessarily cause an issue, the unintended consequence is that virtual hub is connected at USB full speed rather than high speed, so devices 6 and higher are limited to 12 Mbps.

The first alternative we tried was to pass the whole USB root through using VT-d/IOMMU. However, of our two hardware variants, the first doesn't support IOMMU while the second has the USB and our NICs in the same IOMMU group, so while the passthrough works great for USB, we lose our networking :-(

The second method we've tried is to add a second USB controller, like this:

qm set 100 -args "-device usb-ehci,id=ehci,addr=0x5 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.1 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.3 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.4 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.5 \
-device usb-host,bus=ehci.0,hostbus=1,hostport=7.7 \
-device nec-usb-xhci,id=xhci,addr=0x6 \
-device usb-host,bus=xhci.0,hostbus=1,hostport=7.2 \
-device usb-host,bus=xhci.0,hostbus=1,hostport=7.6 \
-device usb-host,bus=xhci.0,hostbus=1,hostport=7.8 \
-device usb-host,bus=xhci.0,hostbus=1,hostport=7.9"

You'll see we've had to add the second group of devices using the XHCI controller, it's apparently possible to add more than one EHCI by using a different id but startup wasn't always reliable for us (note: in PVE 5.0 you can use the qemu-xhci USB device from QEMU 2.9). However, XHCI just hasn't proved stable for us in our use case yet, so we do need to revisit this one at a later date. We definitely see less latency in the devices though, so XHCI is the way forward, they just reset after a short time though.

As an out of the box idea, we even tried experimenting with USB/IP, adding devices to the USB/IP server in the host and creating a VHCI bus in the VM, using a dedicated bridge. Results were... well... mixed

Passing the entire hub through seems like the best option - it would be great for us too. However, after a lot of digging around lists and docs, I don't think it's as clear cut as that. You are reusing the same code that connects the virtual HID devices such as pointer/tablet and the like so you're actually connecting your USB devices to this same controller rather than "passing them through" to the OS. Multifunction devices can have one driver on the VM, while leaving the other device visible on the host. You can read more on QEMU USB at https://github.com/qemu/qemu/blob/master/docs/usb2.txt

I'm sure once we get back to testing, we'll find ways of working around lot of these quirks but for now for our use case, EHCI is the most stable. One of the other methods might work well for someone else though.

 

[krikey]

Im a bit late to the party on this one, but im running PVE 6.4.x and have assigned 9 USB devices after adjusting the /usr/share/perl5/PVE/QemuServer.pm file as follows:

my $MAX_USB_DEVICES = 12;

The VM seems to see all the USB devices, but the PVE GUI only reports the first 5. I presume that the GUI is using some other method for limiting access to allocating or displaying any USB devices that have been allocated to a VM by editing the .conf file(s) directly?

 

[Andreas S.]

Some years have passed and Proxmox 7 is the current version now.

Just to give an update, this is what has worked for me so far:

Figure out the USB Device bus and port ids:

# lsusb -t

....

# lsusb

Add it to the VM configuration

# cd /etc/pve/qemu-server
# cp 110.conf 110.conf.bak
# qm set 110 -args "\
-device qemu-xhci,id=xhci1 \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=1.1,id=Device_1 \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=1.2,id=Device_2 \
-device usb-host,bus=xhci1.0,hostbus=3,hostport=1.3,id=Device_3 \
-device qemu-xhci,id=xhci2 \
-device usb-host,bus=xhci2.0,hostbus=7,hostport=1,id=Device_14 \
-device usb-host,bus=xhci2.0,hostbus=7,hostport=3,id=Device_5 \
-device usb-host,bus=xhci2.0,hostbus=7,hostport=4.3.1,id=Device_6 \
"
# qm start 110

# qm monitor 110
qm> info usb
  Device 0.1, Port 1, Speed 480 Mb/s, Product QEMU USB Tablet, ID: tablet
  Device 2.1, Port 1, Speed 480 Mb/s, Product Device_1, ID: Device_1
  Device 2.2, Port 2, Speed 480 Mb/s, Product Device_2, ID: Device_2
  Device 2.3, Port 3, Speed 12 Mb/s, Product Device_3, ID: Device_3
  Device 3.1, Port 1, Speed 12 Mb/s, Product Device_4, ID: Device_4
  Device 3.2, Port 2, Speed 1.5 Mb/s, Product USB Keyboard, ID: Device_5
  Device 3.3, Port 3, Speed 12 Mb/s, Product Apple Keyboard, ID: Device_6
qm> quit

Is there any way to move the QEMU USB Tablet to the end of the list?

I really doubt it needs the 480 Mb/s. Would be nice to be able to use it for something else...

 

[geraldcl]

Hello, any update on this topic? Why so difficult to add more than 5 USB? I have the latest version of Proxmox installed and I can't add 15 USB drives to a virtual machine. Does anyone have information on how to do it. Thanks.

 

[matrix1999]

Im a bit late to the party on this one, but im running PVE 6.4.x and have assigned 9 USB devices after adjusting the /usr/share/perl5/PVE/QemuServer.pm file as follows:

my $MAX_USB_DEVICES = 12;

The VM seems to see all the USB devices, but the PVE GUI only reports the first 5. I presume that the GUI is using some other method for limiting access to allocating or displaying any USB devices that have been allocated to a VM by editing the .conf file(s) directly?

Hi there, I can't seem to find the declaration using Proxmox 8. Do you happen to know where this is now set?

/usr/share/perl5/PVE/QemuServer.pm

 

[bestho]

Hello @matrix1999 ,

you have found a solution?

 

[fiona]

Hi,
the limit is 14 nowadays, but you need to select a machine version >= 7.1 (in the VM's Hardware tab) (and in case of Windows, have Windows 8 or newer). Be careful however, as Windows can be very sensible to virtual hardware layout changes: https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_system_settings

 

[matrix1999]

Hello @matrix1999 ,

you have found a solution?

Not really. What I ended up doing is to pass through the whole USB controller to the VM. I know this may not be ideal for everyone. However, it works for my use case.

Also, it is worth noting that some PC/Server has multiple USB controllers. In that case you may only need to pass through one (or however many of them), and you just have to make sure you pass through the right one.