Random disconnection USB3 and/or NVME HDD containing the Host (PVE 8.1.10)

Also, I found this in your 5/10 post:

Mar 29 07:21:10 pve kernel: usb: port power management may be unreliable


https://unix.stackexchange.com/ques...er-management-may-be-unreliable-actually-mean

https://www.kernel.org/doc/html/v4.16/driver-api/usb/power-management.html


Look into the power saving stuff in your BIOS, and you may have to echo some stuff into /sys/bus to prevent the USB ports from getting powerdown "savings". From what I saw you have a crazy amount of usb ports

Mar 29 07:21:10 pve kernel: hub 1-0:1.0: 16 ports detected
Mar 29 07:21:10 pve kernel: hub 2-0:1.0: 10 ports detected
 
  • Like
Reactions: justinclift
Thank you for your help, the HP micro server was purchased on Amazon Renewed and is guaranteed for another two months (yes, it's reconditioned) but personally, I really don't think the problem is hardware.

As for the PCIe/USB3 adapter, it's a mini PC, there is no second PCe port for data.

HP EliteDesk 800 G3 Desktop Mini Business PC
  1. 1 Turbo drive (M.2 PCIe) M.2 PCIe x1-2230 (for WLAN)
  2. 1 Turbo drive (M.2 PCIe) M.2 PCIe x4-2280 (for storage)

For the suggestion to upgrade to 2.5Gb, my NAS only has one 1GB Ethernet port and all these other USB ports are occupied and I'm not even sure it would accept a USB3/2.5GB adapter. This isn't a solution I'm going to pursue, but thanks for the suggestion.

Disabling power management is one of the first things I disabled in the BIOS during the initial install, but I'll double-check by connecting an external display, maybe I missed a little something else. Thank you for this reminder.

What I don't understand is that the hard drive is in use when it disconnects, if power management was enabled it should not disconnect the hard drive (Frigate reads and writes constantly).

Also, autosuspend is disabled by default in the kernel since 2.6, from what I understand, so it doesn't come from there (I don't use a HUB).

https://www.kernel.org/doc/html/v6.5/driver-api/usb/power-management.html

When in doubt, In order to completely disable LMP USB3 after reading this topic

https://askubuntu.com/questions/802712/how-to-disable-usb3-lpm-hardware-controlled-power-management

I just updated the kernel to version 6.5.13-5-pve.

Code: 
root@pve:~# pveversion
pve-manager/8.1.10/4b06efb5db453f29 (running kernel: 6.5.13-5-pve)

I also made these changes in GRUB

Code: 
nano /etc/default/grub
Replace
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
by
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on usbcore.quirks=0480:a200:k,18d1:9302,k"
update-grub
reboot

After:
Code: 
root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.13-5-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on

Before reboot:
Code: 
root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.13-5-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on usbcore.quirks=0480:a200:k,18d1:9302,k

I tried these commands before but it doesn't work!

Code: 
root@pve:~# echo "0" > /sys/bus/usb/devices/2-3:1.0/power/usb3_hardware_lpm_u1
-bash: /sys/bus/usb/devices/2-3:1.0/power/usb3_hardware_lpm_u1: Permission denied

root@pve:~# echo "0" > /sys/bus/usb/devices/2-3:1.0/power/usb3_hardware_lpm_u2
-bash: /sys/bus/usb/devices/2-3:1.0/power/usb3_hardware_lpm_u2: Permission denied


After :
Code: 
root@pve:~# find /sys -iname "*usb3*"
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/power/usb3_hardware_lpm_u2
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/power/usb3_hardware_lpm_u1
/sys/devices/pci0000:00/0000:00:14.0/usb2/power/usb3_hardware_lpm_u2
/sys/devices/pci0000:00/0000:00:14.0/usb2/power/usb3_hardware_lpm_u1
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port9/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port7/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port5/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port3/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port1/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port8/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port6/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port10/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port4/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port2/usb3_lpm_permit

Before :

Code: 
root@pve:~# find /sys -iname "*usb3*"
/sys/devices/pci0000:00/0000:00:14.0/usb2/power/usb3_hardware_lpm_u2
/sys/devices/pci0000:00/0000:00:14.0/usb2/power/usb3_hardware_lpm_u1
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port9/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port7/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port5/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port3/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port1/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port8/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port6/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port10/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port4/usb3_lpm_permit
/sys/devices/pci0000:00/0000:00:14.0/usb2/2-0:1.0/usb2-port2/usb3_lpm_permit

Code: 
root@pve:~# lsusb
Bus 002 Device 002: ID 18d1:9302 Google Inc.
Bus 002 Device 003: ID 0480:a200 Toshiba America Inc External USB 3.0
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Code: 
root@pve:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
    |__ Port 7: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M


Code: 
root@pve:~# dmesg | grep "usb"
[    0.313114] usbcore: registered new interface driver usbfs
[    0.313114] usbcore: registered new interface driver hub
[    0.313114] usbcore: registered new device driver usb
[    1.549408] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.05
[    1.549412] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.549413] usb usb1: Product: xHCI Host Controller
[    1.549415] usb usb1: Manufacturer: Linux 6.5.13-5-pve xhci-hcd
[    1.549417] usb usb1: SerialNumber: 0000:00:14.0
[    1.553458] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.05
[    1.553462] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.553464] usb usb2: Product: xHCI Host Controller
[    1.553466] usb usb2: Manufacturer: Linux 6.5.13-5-pve xhci-hcd
[    1.553468] usb usb2: SerialNumber: 0000:00:14.0
[    1.556192] usb: port power management may be unreliable
[    1.896617] usb 2-7: new SuperSpeed USB device number 2 using xhci_hcd
[    1.917101] usb 2-7: LPM exit latency is zeroed, disabling LPM.
[    1.917347] usb 2-7: New USB device found, idVendor=18d1, idProduct=9302, bcdDevice= 1.00
[    1.917350] usb 2-7: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    2.188585] usb 2-3: new SuperSpeed USB device number 3 using xhci_hcd
[    2.209473] usb 2-3: New USB device found, idVendor=0480, idProduct=a200, bcdDevice= 0.00
[    2.209477] usb 2-3: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[    2.209479] usb 2-3: Product: External USB 3.0
[    2.209481] usb 2-3: Manufacturer: TOSHIBA
[    2.209483] usb 2-3: SerialNumber: 20150911027639F
[    2.213957] usb-storage 2-3:1.0: USB Mass Storage device detected
[    2.214204] scsi host2: usb-storage 2-3:1.0
[    2.214277] usbcore: registered new interface driver usb-storage
[    2.215425] usbcore: registered new interface driver uas
[   22.112788] usb 2-7: reset SuperSpeed USB device number 2 using xhci_hcd
[   22.132827] usb 2-7: LPM exit latency is zeroed, disabling LPM.
 
Last edited:
I had already read this article and from what I understood to the contrary, this activates automatic deactivation which is disabled by default.

You have to set 0 to deactivate it but it doesn't work

Code: 
root@pve:~# echo "0" > "/sys/bus/usb/devices/2-3:1.0/power/autosuspend_delay_ms"
-bash: /sys/bus/usb/devices/2-3:1.0/power/autosuspend_delay_ms: Permission denied

Code: 
root@pve:~# cat /sys/bus/usb/devices/2-3:1.0/power/control
cat: '/sys/bus/usb/devices/2-3:1.0/power/control': No such file or directory

In the meantime, I just connected an external screen to control the BIOS, the power saving function was all disabled.
I took the opportunity to update the BIOS, you never know, it went from version v02.47 to v02.49.

I will now wait a few days to see how the system behaves before continuing to make any further modifications.
Thanks for your help.
 
  • Like
Reactions: Kingneutron
After 4 days and 11 hours, I just had a disconnection / reconnection of the CORAL USB TPU (not the HDD), maybe this will happen in 2 days, to see...

Logs PVE
Code: 
Apr 26 04:41:59 pve kernel: usb 2-7: reset SuperSpeed USB device number 2 using xhci_hcd
Apr 26 04:41:59 pve kernel: usb 2-7: LPM exit latency is zeroed, disabling LPM.

Code: 
Apr 26 04:20:13 pve kernel: eth0: left promiscuous mode
Apr 26 04:24:13 pve pvedaemon[4098105]: worker exit
Apr 26 04:24:13 pve pvedaemon[1037]: worker 4098105 finished
Apr 26 04:24:13 pve pvedaemon[1037]: starting 1 worker(s)
Apr 26 04:24:13 pve pvedaemon[1037]: worker 19614 started
Apr 26 04:25:02 pve kernel: eth0: entered promiscuous mode
Apr 26 04:25:13 pve kernel: eth0: left promiscuous mode
Apr 26 04:30:02 pve kernel: eth0: entered promiscuous mode
Apr 26 04:30:13 pve kernel: eth0: left promiscuous mode
Apr 26 04:35:02 pve kernel: eth0: entered promiscuous mode
Apr 26 04:35:13 pve kernel: eth0: left promiscuous mode
Apr 26 04:40:02 pve kernel: eth0: entered promiscuous mode
Apr 26 04:40:13 pve kernel: eth0: left promiscuous mode
Apr 26 04:41:59 pve kernel: usb 2-7: reset SuperSpeed USB device number 2 using xhci_hcd
Apr 26 04:41:59 pve kernel: usb 2-7: LPM exit latency is zeroed, disabling LPM.
Apr 26 04:45:01 pve kernel: eth0: entered promiscuous mode
Apr 26 04:45:12 pve kernel: eth0: left promiscuous mode
Apr 26 04:48:06 pve pveproxy[1044]: worker 4170479 finished
Apr 26 04:48:06 pve pveproxy[1044]: starting 1 worker(s)
Apr 26 04:48:06 pve pveproxy[1044]: worker 34573 started
Apr 26 04:48:06 pve pveproxy[34569]: got inotify poll request in wrong process - disabling inotify
Apr 26 04:48:07 pve pveproxy[34569]: worker exit
Apr 26 04:50:01 pve kernel: eth0: entered promiscuous mode
Apr 26 04:50:12 pve kernel: eth0: left promiscuous mode

Log Frigate (Docker LXC) :

Code: 
2024-04-25 21:33:59.274380637  [2024-04-25 21:33:56] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-25 21:33:59.283484123  [2024-04-25 21:33:59] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-04-26 04:41:27.907808735  [2024-04-26 04:41:27] frigate.watchdog        INFO    : Detection appears to be stuck. Restarting detection process...
2024-04-26 04:41:27.907960279  [2024-04-26 04:41:27] root                              INFO    : Waiting for detection process to exit gracefully...
2024-04-26 04:41:57.937930356  [2024-04-26 04:41:57] root                              INFO    : Detection process didnt exit. Force killing...
2024-04-26 04:41:57.945545414  [2024-04-26 04:41:57] root                              INFO    : Detection process has exited...
2024-04-26 04:41:58.028638048  [2024-04-26 04:41:58] detector.coral            INFO    : Starting detection process: 76072
2024-04-26 04:42:00.883613744  [2024-04-26 04:41:58] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2024-04-26 04:42:00.893758110  [2024-04-26 04:42:00] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

In this case, the GOOGLE CORAL USB TPU reconnects automatically, so it is not a problem, but that will not be the case for the Hard Drive which will not automatically reconnect in reading writing...

;)
 
Last edited:
I have just gone 15 days without disconnecting the hard drive!!!
Updating BIOS and/or kernel 6.5.13-5-pve and/or changes in GRUB must have helped.

I keep crossing my fingers :cool:
 
I had a similar issue where one of my NVMe drives randomly disconnects. I have a pair running zfs for my Proxmox installation and running VMs from.

This problem has been going on for weeks if not months. It wasn't until recently that I saw there was an article and someone suggested to append

Code: 
nvme_core.default_ps_max_latency_us=0
at the end of the GRUB file where root=.... (in my case I modified /etc/kernel/cmdline because I run zfs)

Since this modification, I have been running the system now for well over a week without issue.
 
  • Like
Reactions: jvanoli and justinclift
Loupiouk said:
I really don't think the problem is hardware
Click to expand...
Is there any chance it's more of a hardware problem of the "this hardware limits my options" variety rather than plain faulty hardware?

Asking because there are 2nd hand tower servers from HP, Dell, (etc) that have a bunch of drive bays, network ports, cpu, memory (etc) that will likely do everything you want reliably.

They're not as small and power efficient as the existing stuff you're trying to make work though. ;)
 
@matrix1999 This confirms my tests, since I modified GRUB, I have no longer had a USB disconnection, it will soon be two months. :)
On the other hand, I stopped the PVE updates, it works well like that :)


@justinclift Perhaps with a SATA disk I would not have had this problem, but I had a used (soldered) 2TB USB disk available.
But it's not certain because the CORAL USB also disconnected, in short, for the moment everything works ;)
 
Last edited:
  • Like
Reactions: justinclift
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back