usb disk disconnecting after upgrade to promox 8

[katbyte]

After updating to promox 8 a NVME USB disk with a single VM disk on it which had been working fine for a year has started to disconnect and freeze.

Jun 30 09:02:31 mini kernel: scsi host0: uas_eh_device_reset_handler start
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#16 uas_zap_pending 0 uas-tag 1 inflight: CMD
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#16 CDB: Read(10) 28 00 00 00 00 00 00 01 00 00
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#17 uas_zap_pending 0 uas-tag 2 inflight: CMD
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#17 CDB: Read(10) 28 00 4d 16 5a f0 00 00 40 00
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#18 uas_zap_pending 0 uas-tag 3 inflight: CMD
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#18 CDB: Read(10) 28 00 51 36 6a e0 00 00 20 00
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#19 uas_zap_pending 0 uas-tag 4 inflight: CMD
Jun 30 09:02:31 mini kernel: sd 0:0:0:0: [sda] tag#19 CDB: Read(10) 28 00 4d 16 68 10 00 00 c0 00
Jun 30 09:02:31 mini kernel: usb 4-2: reset SuperSpeed Plus Gen 2x1 USB device number 2 using xhci_hcd
Jun 30 09:02:35 mini kernel: usb 4-2: Enable of device-initiated U1 failed.
Jun 30 09:02:35 mini kernel: usb 4-2: Enable of device-initiated U2 failed.
Jun 30 09:02:35 mini kernel: scsi host0: uas_eh_device_reset_handler success
Jun 30 09:04:20 mini kernel: usb 4-2: Enable of device-initiated U1 failed.
Jun 30 09:04:20 mini kernel: usb 4-2: Enable of device-initiated U2 failed.
Jun 30 09:04:20 mini kernel: scsi host0: uas_eh_device_reset_handler success
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#12 timing out command, waited 180s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=208s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#12 CDB: Read(10) 28 00 51 36 6a e0 00 00 20 00
Jun 30 09:04:20 mini kernel: I/O error, dev sda, sector 1362520800 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 2
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#13 timing out command, waited 180s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=208s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#13 CDB: Read(10) 28 00 4d 16 5a f0 00 00 40 00
Jun 30 09:04:20 mini kernel: I/O error, dev sda, sector 1293310704 op 0x0:(READ) flags 0x0 phys_seg 8 prio class 2
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#14 timing out command, waited 180s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=208s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 00 00 00 00 00 01 00 00
Jun 30 09:04:20 mini kernel: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 2
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#15 timing out command, waited 180s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#15 FAILED Result: hostbyte=DID_RESET driverbyte=DRIVER_OK cmd_age=208s
Jun 30 09:04:20 mini kernel: sd 0:0:0:0: [sda] tag#15 CDB: Read(10) 28 00 4d 16 68 10 00 00 c0 00
Jun 30 09:04:20 mini kernel: I/O error, dev sda, sector 1293314064 op 0x0:(READ) flags 0x0 phys_seg 24 prio class 2

nothing else had changed aside from updating promox (7.4 -> 8). once I moved the VM disk to network storage the machine stopped freezing.

 

[alex#]

Does the problem arise randomly or only on a high/specific workload?

Did you check things like the physical connection, temperature and possible power supply issues?

Is your log from the guest or the host? Please also post the missing one.

Which guest OS do you run and is it up to date?

What's the output from cat /etc/pve/storage.cfg and pveversion --verbose ?

Please provide the VM config when the NVMe is attached/configured again: qm pending <vmid>

Are there any firmware updates available (always back up first)? Depending on how you can access the NVMe via your USB connection, there is a tool called 'nvme-cli' in the repository respectively https://github.com/linux-nvme/nvme-cli .

To get some details of your NVMe please post: nvme list

Maybe that's also something you can look at: https://forum.proxmox.com/threads/p...-0xffffffff-pci_status-0x10.88604/post-490488

 

[katbyte]

I wouldn't say the workload is high, light use ~10-20mbyte sec, but it has been constant over the last year so nothing out of the ordinary was going on when it crashed, and it happened ~5 times. again this config was fine without incident for a year until now and the only thing that changed was upgrading to proxmox 8

connections/temp/power hasn't changed so afaik still all good

log was from host, guest just showed the disk being removed, guest is debian.

guest is up to date as is firmware

[09:52:52] root@mini:~# cat /etc/pve/storage.cfg dir: local path /var/lib/vz content images,snippets,backup,iso,rootdir,vztmpl prune-backups keep-all=1 pbs: backup datastore data server 10.0.0.111 content backup fingerprint a7:27:2e:e8:ca:09:ad:16:34:09:c3:a9:f8:22:c4:e1:e6:27:c8:cb:38:ad:b5:d9:04:f3:01:ab:55:83:1d:de prune-backups keep-all=1 username root@pam lvm: usb vgname usb content rootdir,images nodes mini shared 0 cifs: data path /mnt/pve/data server 10.0.0.5 share zproxmox content vztmpl,rootdir,backup,iso,snippets,images prune-backups keep-last=8,keep-monthly=12,keep-weekly=8 username proxmox [09:52:53] root@mini:~# [09:52:53] root@mini:~# pveversion --verbose proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve) pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916) pve-kernel-6.2: 8.0.2 pve-kernel-5.15: 7.4-4 pve-kernel-6.2.16-3-pve: 6.2.16-3 pve-kernel-5.19.17-2-pve: 5.19.17-2 pve-kernel-5.15.108-1-pve: 5.15.108-1 ceph-fuse: 16.2.11+ds-2 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown: residual config ifupdown2: 3.2.0-1+pmx2 libjs-extjs: 7.0.0-3 libknet1: 1.25-pve1 libproxmox-acme-perl: 1.4.6 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.0 libpve-access-control: 8.0.3 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.0.5 libpve-guest-common-perl: 5.0.3 libpve-http-server-perl: 5.0.3 libpve-rs-perl: 0.8.3 libpve-storage-perl: 8.0.1 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-2 proxmox-backup-client: 2.99.0-1 proxmox-backup-file-restore: 2.99.0-1 proxmox-kernel-helper: 8.0.2 proxmox-mail-forward: 0.2.0 proxmox-mini-journalreader: 1.4.0 proxmox-offline-mirror-helper: 0.6.1 proxmox-widget-toolkit: 4.0.5 pve-cluster: 8.0.1 pve-container: 5.0.4 pve-docs: 8.0.3 pve-edk2-firmware: 3.20230228-4 pve-firewall: 5.0.2 pve-firmware: 3.7-1 pve-ha-manager: 4.0.2 pve-i18n: 3.0.4 pve-qemu-kvm: 8.0.2-3 pve-xtermjs: 4.16.0-3 qemu-server: 8.0.6 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.1.12-pve1 [09:53:21] root@mini:~#

the disk is not showing up with that name tool

[09:54:29] root@mini:~# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 /dev/ng0n1 C079254002CJRR7AW APPLE SSD AP0512M 1 500.28 GB / 500.28 GB 4 KiB + 0 B 1274.100 [09:54:31] root@mini:~#

I presume because its usb

[09:55:10] root@mini:~# lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 002: ID 0bda:9210 Realtek Semiconductor Corp. RTL9210 M.2 NVME Adapter Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub [09:55:19] root@mini:~#

vm config

[09:56:18] root@mini:~# qm pending 108 cur agent: 1 cur boot: order=scsi0;ide2;net0 cur cores: 12 cur cpu: host cur description: scsi1%3A usb%3Avm-108-disk-0,backup=0,cache=writeback,discard=on,iothread=1,size=1850G,ssd=1%0Ascsi1%3A usb%3Avm-108-disk-0,backup=0,cache=writeback,discard=on,iothread=1,size=1850G,ssd=1 cur ide2: none,media=cdrom cur machine: q35 cur memory: 8192 cur meta: creation-qemu=7.0.0,ctime=1668583708 cur name: seed cur net0: virtio=7A:DE:D3:93:66:46,bridge=vmbr1,firewall=1,mtu=9000,queues=8 cur numa: 0 cur onboot: 1 cur ostype: l26 cur scsi0: local:108/vm-108-disk-0.qcow2,discard=on,iothread=1,size=32G cur scsi1: usb:vm-108-disk-0,backup=0,discard=on,iothread=1,size=32G cur scsihw: virtio-scsi-single cur smbios1: uuid=46f4135e-7c03-4d27-8b95-ef16344ac085 cur sockets: 1 cur vmgenid: 2e565f48-4a6f-4391-b05c-7e174ac818bb [09:56:23] root@mini:~#

 

[alex#]

Perhaps you can try to narrow down the problem by booting with an earlier kernel version (https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_kernel_pin).

 

[katbyte]

I didn't have time to really debug it so I just removed the USB disk and moved to network storage