[SOLVED] USB Ports not working after Proxmox update

[andre85]

Hi there.

I've updated my proxmox (apt update && apt full-upgrade).
After the reboot, my USB ports of my Asrock N100DC-ITX are partly not working. To be more specific, these ports + my front USB stopped working:


I can't 100% confirm that it's related to the update, since I didn't immediately restart my server, but I can't think off what else could be the cause of this.
Any idea how I can debuff this further? I'm no linux expert, so I'm not really sure

Thank you in advance!
André

 

[leesteken]

It's probably the USB controller not playing nice with the iOMMU. Try adding intel_iommu=off to your kernel parameters (it was turned on by default by Ubuntu/Proxmox) in kernel 6.8.

 

[gfngfn256]

I can only link a problem I appear to have here, although in my case everything works, until I start copying some files & only then my WHOLE USB bus fails. Also on my system (not similar I believe to yours), if I use a USB flash drive (non UAS) as opposed to a SATA to USB (UAS) I have no problems. So I'm not sure we are related. My problem also only started after recent upgrades.

 

[andre85]

It's probably the USB controller not playing nice with the iOMMU. Try adding intel_iommu=off to your kernel parameters (it was turned on by default by Ubuntu/Proxmox) in kernel 6.8.

Not sure if I did this correctly. I've nanoed /etc/default/grub and added

intel_iommu=off

followed by update-grub
Because it didn't detect my usb pendrive I've also rebooted, but still no luck unfortunately. Did I do something wrong?

I can only link a problem I appear to have here, although in my case everything works, until I start copying some files & only then my WHOLE USB bus fails. Also on my system (not similar I believe to yours), if I use a USB flash drive (non UAS) as opposed to a SATA to USB (UAS) I have no problems. So I'm not sure we are related. My problem also only started after recent upgrades.

I don't think that we have the same issue, because mine doesn't detect anything unfortunately.

 

[leesteken]

Not sure if I did this correctly. I've nanoed /etc/default/grub followed by update-grub
Because it didn't detect my usb pendrive I've also rebooted, but still no luck unfortunately. Did I do something wrong?

I can't tell from the little information here, nor do I know for sure that this is the problem/fix. Which bootloader is your Proxmox using? What is the output of cat /proc/cmdline ?

 

[andre85]

I can't tell from the little information here, nor do I know for sure that this is the problem/fix. Which bootloader is your Proxmox using? What is the output of cat /proc/cmdline ?

This is the output for both:

root@homeserver:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet

root@homeserver:~# efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0004,0002,0000
Boot0000* Wind￿￿s Boot Manager  VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)WINDOWS.........x...B.C.D.O.B.J.E.C.T.=.{.9.d.e.a.8.6.2.c.-.5.c.d.d.-.4.e.7.0.-.a.c.c.1.-.f.3.2.b.3.4.4.d.4.7.9.5.}...a................
Boot0001* proxmox       HD(2,GPT,41b34d87-eea8-4f30-a5a6-969549f4c0cc,0x800,0x200000)/File(\EFI\proxmox\shimx64.efi)
Boot0002* debian        VenHw(99e275e7-75a0-4b37-a2e6-c5385e6c00cb)
Boot0004* UEFI OS       HD(2,GPT,41b34d87-eea8-4f30-a5a6-969549f4c0cc,0x800,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

Sorry if I didn't provide enough information, but I'm really not that experienced with that and not sure what else I can/should provide.

 

[leesteken]

root@homeserver:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet

As you can see intel_iommu=off is not there (and it probably looks nothing like you saw in /etc/default/grub). You probably need to edit the other bootloader (systemd-boot) in /etc/kernel/cmdline (everything on the single first line, no additional lines): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline

 

[andre85]

As you can see intel_iommu=off is not there (and it probably looks nothing like you saw in /etc/default/grub). You probably need to edit the other bootloader (systemd-boot) in /etc/kernel/cmdline (everything on the single first line, no additional lines): https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_edit_kernel_cmdline

Ah I've found it. The /etc/kernel/cmdline was completely empty. But thanks to this thread i figured out that I had to add it to GRUB_CMDLINE_LINUX_DEFAULT rather than pasting it just at the end of the file.
Now my bootloader looks like this:

root@homeserver:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet intel_iommu=off

Unfortunately this didn't fix the USB though

 

[andre85]

I've just checked dmesg -w if the insertion / pulling the usb pen drive shows anything.
While for the first 4 ports it's detecting the pen drive

[ 1709.529130] usb 3-3: new high-speed USB device number 6 using xhci_hcd
[ 1709.682491] usb 3-3: New USB device found, idVendor=0781, idProduct=5567, bcdDevice= 1.00
[ 1709.682511] usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1709.682518] usb 3-3: Product: Cruzer Blade
[ 1709.682523] usb 3-3: Manufacturer: SanDisk
[ 1709.682528] usb 3-3: SerialNumber: 4C530000100509114060
[ 1709.685202] usb-storage 3-3:1.0: USB Mass Storage device detected
[ 1709.687962] scsi host3: usb-storage 3-3:1.0
[ 1710.714529] scsi 3:0:0:0: Direct-Access     SanDisk  Cruzer Blade     1.00 PQ: 0 ANSI: 6
[ 1710.717562] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 1710.718613] sd 3:0:0:0: [sdd] 250085376 512-byte logical blocks: (128 GB/119 GiB)
[ 1710.720256] sd 3:0:0:0: [sdd] Write Protect is off
[ 1710.720271] sd 3:0:0:0: [sdd] Mode Sense: 43 00 00 00
[ 1710.721351] sd 3:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 1710.750913]  sdd: sdd1
[ 1710.751171] sd 3:0:0:0: [sdd] Attached SCSI removable disk
[ 1730.781408] usb 3-3: USB disconnect, device number 6

it doesn't show anything for the front USB nor the two marked from post 1.
It really looks like the USB hub is gone or something

 

[leesteken]

I've once had a USB TV tuner break all the USB ports on a motherboard. Does the motherboard BIOS see devices connected to those ports?
Maybe boot a live Linux CD (like GParted Live or Ubuntu installer) and see if the USB ports work if you are not using Proxmox.

 

[andre85]

Hmpf, you may be right...
Just connected my keyboard to one of the ports - it doesn't get power, nor did I manage to get into the BIOS using this port. So it really looks like the ports are dead :\

Looks like I'm gonna RMA the board

 

[gfngfn256]

Hmpf, you may be right...
Just connected my keyboard to one of the ports - it doesn't get power, nor did I manage to get into the BIOS using this port. So it really looks like the ports are dead :\

Looks like I'm gonna RMA the board

Interesting. Did all the said ports work - before PVE update?

 

[andre85]

Yes. I was using my sonoff zigbee adapter on one of the ports.
And the BT adapter was connected to the front USB.

 

[gfngfn256]

Did you try shutting down machine - remove all power - then restart & try port/s?

 

[andre85]

Did you try shutting down machine - remove all power - then restart & try port/s?

This actually seems like it has fixed the issue!
I've tried shutting it down (without removing power), but this didn't help. I've now removed power for like a minute, turned back on and it seems like the ports are working again

Thanks a ton (to all!)!

 

[gfngfn256]

Happy you got it settled. Maybe mark this thread as [SOLVED]
( right-hand side under thread title).

 

[leesteken]

This actually seems like it has fixed the issue!
I've tried shutting it down (without removing power), but this didn't help. I've now removed power for like a minute, turned back on and it seems like the ports are working again

Then it probably was a short-circuit, ground-fault or overload protection that got triggered. Thank you @gfngfn256!