PVE doing USB 2 speeds while HDD is connected via USB 3.2 gen 2

[B4ms]

Hello,

I hope someone can help me figure this out - searching for several days now but to no avail.

I'm running Proxmox on an intel NUC, specifically the NUC13ANKi3. Connected to this I have a ICY BOX IB-RD3621-C31 disk array, containing one Toshiba MG Series 3.5" 20000 GB SATA.

The disk array is connected to the NUC via USB-C. The ICY BOX has a 3.2 Gen 2 USB-port and I connect this to the USB-C port of the NUC (which is USB 4).
I would expect that since the NUC USB-C port is USB 4, the ICY BOX is the 'bottleneck' with its 3.2 Gen 2.

When running iotop while writing a large file, I noticed speeds of max 40mb/s however. This got me searching;

When running lsusb -t I'm seeing:

lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/3p, 20000M/x2
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 5: Dev 3, If 0, Class=Mass Storage, Driver=uas, 480M
|__ Port 10: Dev 2, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 10: Dev 2, If 1, Class=Wireless, Driver=btusb, 12M

The Mass Storage device is running 480M, which is USB 2 speeds. (Mind that I'm running these commands on the PVE shell itself, so not in a VM with passthrough - although the drive is passed through using qm set 102 --scsi1 /dev/disk/by-id/ata-TOSHIBA_MG10ACA20TE_Z2C0A3LUF4MJ)

I'm lost at how to get the USB 3 speeds working. Could this be a driver issue (although the kernel driver in use seems to be xhci_hcd - and the Driver is "uas")?

lspci -v | grep xhci
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

For reference, I used the following steps to add the USB drive to Proxmox after install;

lsblk
parted /dev/sda mklabel gpt
parted -a opt /dev/sda mkpart primary ext4 0% 100%
mkfs.ext4 -L StorageOne /dev/sda1
nano /etc/fstab
LABEL=StorageOne /mnt/data ext4 defaults 0 2
mount -a
systemctl daemon-reload
qm set 102 --scsi1 /dev/disk/by-id/ata-TOSHIBA_MG10ACA20TE_Z2C0A3LUF4MJ

Did I perhaps make a mistake in how the drive was partitioned? Is it a driver issue probably? Should I enable something somewhere? Should I disable something somewhere? Any pointers are very much welcome. Thanks in advance!

 

[Bent]

There is loads of information in your post, regarding your reference what you did after installing Proxmox:

As far as I can guess, you did partition, format and then mount your external disk to your Proxmox host. After mounting the disk you passed the device through to your virtual machine, while the disk is still mounted to the Proxmox host?


Additionally, what type of cable do you use to connect your external storage to the system itself? Not all USB-C cables are equal, some only support USB 2.0 speeds (!).

 

[B4ms]

There is loads of information in your post, regarding your reference what you did after installing Proxmox:

As far as I can guess, you did partition, format and then mount your external disk to your Proxmox host. After mounting the disk you passed the device through to your virtual machine, while the disk is still mounted to the Proxmox host?


Additionally, what type of cable do you use to connect your external storage to the system itself? Not all USB-C cables are equal, some only support USB 2.0 speeds (!).

Thanks for the fast reply @Bent . With regards to the USB-C cable, I'm using the one that came with the ICY BOX - I'd be surprised if they'd provide a USB 2 cable when advertising USB 3.2 Gen 2 speeds (but perhaps that is the problem - I'd have to check with a different cable, but first am curious as to the below: ).

And yes, I have partitioned, formated and then mounted the external disk to Proxmox on my PVE shell. And then ran qm set 102 etc. to give my VM access to it. I'm quite new at all this and am leaning heavily on blogs and forum posts. I found that this was the way to do it. Your response suggests there is another (better?) way?

Am I mistaken in thinking I cannot use 'qm set' when it is not mounted on the pve? It probably won't make a difference to the USB speeds though, right?

 

[Bent]

For some reason I can't see your reply here but got an email with your text.

My recommendation would be to unmount the disk from your hypervisor (and remove it's entry from fstab to avoid accidental mounting) if it's not used on the HV itself. Otherwise you'll run into concurrent access issues which will result in corrupted data. (ext4 is not recommended to be mounted by multiple systems at once and should even block this)

Additionally the easiest way to passthrough a USB Drive to a VM is via USB passthrough:

 

[B4ms]

Thank you @Bent. I'm posting but the post gets stuck at a moderator approval flow.

I have checked the cable, and it is tagged with a 10GB-label. It's also quite a thick cable, so it looks as if it is supporting those speeds.

I'll follow your steps and try again via the other way of mounting, and report back !

 

[B4ms]

I think that the system was already unmounting the drive from the PVE automatically, as the entry was still there in my fstab file - but it wasn't mounted. Or at least, unmounting it running 'sudo umount /mnt/data' returned that it wasn't mounted. I have removed the line from my fstab file on the PVE. (After a reboot of the PVE, the drive is still functioning and I'm able to mount it via the Add USB option you've provided @Bent .)

I have then removed the (qm set) hardware from the VM using the detach button.

When adding a USB port it shows that the speeds are USB 2 however, confirming that it's somehow not understanding this all is USB 3.2;



When adding it to my VM, and doing ''sudo mount /dev/sdb1 /mnt/data'' it's working as it always has. So that's all good now, and probably less prone to corruption (Thanks @Bent).

However, the question remains; how do I make Proxmox understand that this is actually a USB 3.2 Gen 2 device?

 

[Bent]

This seems like a really weird quirk. USB 4 / TB should be supported in any current PVE kernel. Also USB 4 / TB is definitely backwards compatible.

Can you check if there is any difference when trying either of these three ports?



Port #3 will of course require a suitable USB-C to USB-A Cable which at least supports 5 Gbit/s (should be enough to saturate your HDD still).

Also if any other USB-C to USB-C cable is available please check if this makes a difference, we had quite a few times weird quirks with 'technically correct OEM cables'.

Aside from this the ASM1352R seems to be a really old controller from ~2015. This does not mean it's good or bad but a quick google search indicates quite a few folks with issues on Linux.

Maybe we can find something in the logs:

journalctl --no-pager -b0 --priority err

 

[B4ms]

@Bent , first of all, thanks for all the effort you're taking in helping me.

I've ran the bash as sudo and it returns -- no entries --.

I've tried two other cables (USB-C to USB-C for an iPad Pro and one from a Macbook Pro M1, both being fast charge cables I'm assuming they have at least USB 2 speed capabilities). I've tried both the USB C ports on the back with all three cables and it keeps telling me it's USB 2. The cable that came with the disk array has 10GB logo's on it, so that should count as USB 3, right?

I don't own any C to A cables I think. So I have to trust that the above is sufficient for testing.

I'll dive into the ASM1352R in combination with Linux a bit, perhaps that wil yield a result.

 

[B4ms]

Super strange, I have put the USB-C cable back as it was (the original one that came with the device), and suddenly it is showing as USB 3.1:



However, The Use USB3 option is greyed out. I've tried removing and re-adding, but it stays greyed out. @Bent

Doing lsusb -t is showing 5000M now, 10x as much as before. What is going on?! It seems fixed though.....!

 

[Bent]

Super strange, I have put the USB-C cable back as it was (the original one that came with the device), and suddenly it is showing as USB 3.1:

View attachment 61347

However, The Use USB3 option is greyed out. I've tried removing and re-adding, but it stays greyed out. @Bent

Doing lsusb -t is showing 5000M now, 10x as much as before. What is going on?! It seems fixed though.....!

That's what I meant with 'weird quirk'. We had quite a few of these seemingly 'weird' issues with AS Media controllers in the past.

The fact that USB3 is greyed out (but likely ticked) means it's USB3. You should get good speeds within your VM now. You may want to check if things still work as expected after a complete shutdown and disconnect from power.

 

[B4ms]

Yet another strange development: It suddenly disconnected from the VM - and i can only mount it as a USB 2.0 again...

 

[B4ms]

That's what I meant with 'weird quirk'. We had quite a few of these seemingly 'weird' issues with AS Media controllers in the past.

The fact that USB3 is greyed out (but likely ticked) means it's USB3. You should get good speeds within your VM now. You may want to check if things still work as expected after a complete shutdown and disconnect from power.

So far it's only showing USB 2.0 again, after several powerdowns and reboots. I've tried switching ports again as well, but no 3.1 status.
The USB3 tick was off by the way, I see the screenshot did not show that. But it was greyed out and off.

I guess if there is a faulty / quirky controller in play, I'm out of luck.

Much obliged for the help once more @Bent !

 

[B4ms]

Going down the rabbit hole, apparently there is a fix available but I have no idea if this is something I could implement on my end;

https://marc.info/?l=linux-usb&m=164209631518416&w=2

I've also looked at adding a USB quirk as via this bug report; https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1742318

Also, no luck. So I've removed this quirk again.

Edit:

Found a high speed USB-C to USB-A cable, unfortunately also still USB 2.0 even though it's a different port entirely.

 

[Bent]

Going down the rabbit hole, apparently there is a fix available but I have no idea if this is something I could implement on my end;

https://marc.info/?l=linux-usb&m=164209631518416&w=2

I've also looked at adding a USB quirk as via this bug report; https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1742318

Also, no luck. So I've removed this quirk again.

Edit:

Found a high speed USB-C to USB-A cable, unfortunately also still USB 2.0 even though it's a different port entirely.

Interesting find, you might want to read this on how to report a bug for Proxmox. Maybe Proxmox will add the patch to the PVE Kernel, though I doubt it due to stability reasons. Cheers!

 

[B4ms]

Maybe we can find something in the logs:

journalctl --no-pager -b0 --priority err

Hi again, it's been a while and this issue still persists. I tried to rerun the command you've provided and it now returns the following:

journalctl --no-pager -b0 --priority err
Mar 12 07:37:27 pve kernel: pci 0000:00:07.0: DPC: RP PIO log size 0 is invalid
Mar 12 07:37:27 pve kernel: pci 0000:00:07.2: DPC: RP PIO log size 0 is invalid
Mar 12 07:37:27 pve systemd-modules-load[409]: Failed to find module 'vfio_virqfd'
Mar 12 07:37:28 pve kernel: Bluetooth: hci0: Failed to load Intel firmware file intel/ibt-0040-0041.sfi (-2)
Mar 12 07:37:28 pve kernel: Bluetooth: hci0: Reading supported features failed (-56)
Mar 12 07:37:28 pve kernel: Bluetooth: hci0: Failed to read MSFT supported features (-56)
Mar 12 07:37:28 pve kernel: iwlwifi 0000:00:14.3: WRT: Invalid buffer destination
Mar 12 07:37:30 pve (ometheus)[829]: prometheus.service: Failed at step EXEC spawning /usr/bin/prometheus: No such file or directory
Mar 12 07:37:30 pve systemd[1]: Failed to start openipmi.service - LSB: OpenIPMI Driver init script.
Mar 12 07:37:30 pve (ometheus)[939]: prometheus.service: Failed at step EXEC spawning /usr/bin/prometheus: No such file or directory
Mar 12 07:37:30 pve systemd[1]: Failed to start prometheus-pve-exporter.service - Prometheus exporter for Proxmox VE.
Mar 12 07:37:30 pve (ometheus)[1075]: prometheus.service: Failed at step EXEC spawning /usr/bin/prometheus: No such file or directory
Mar 12 07:37:30 pve (ometheus)[1142]: prometheus.service: Failed at step EXEC spawning /usr/bin/prometheus: No such file or directory
Mar 12 07:37:31 pve (ometheus)[1143]: prometheus.service: Failed at step EXEC spawning /usr/bin/prometheus: No such file or directory
Mar 12 07:37:31 pve systemd[1]: Failed to start prometheus.service - Prometheus.
Mar 12 07:37:33 pve kernel: ucsi_acpi USBC000:00: error -ETIMEDOUT: PPM init failed
Mar 12 07:38:52 pve QEMU[2078]: kvm: vfio-pci: Cannot read device rom at 0000:00:02.0
Mar 12 07:38:52 pve QEMU[2078]: Device option ROM contents are probably invalid (check dmesg).
Mar 12 07:38:52 pve QEMU[2078]: Skip option ROM probe with rombar=0, or load from file with romfile=

Talking to chatgpt it suggests that either the ucsi_acpi error is the culprit - or the openipmi driver init. Does anything above look like it could be the cause of proxmox recognizing my USB 3 disk array (AS Media) as USB 2?

Or should I just accept that this usb controller doesn't fit well with Linux...