USB Drive disappears from lsblk after a few days

T

teeeeee

New Member
Feb 1, 2023
26
2
3
I have a 240GB SSD drive which is connected via USB. After plugging it in, the drive initally shows up correctly in lsblk and fdisk -l, and I am able to mount it without problem on the Proxmox host. The problem is that the drive seems to disappear from the host after a couple of days, and this has happened around 5 times now. When this happens, it does not appear in lsblk anymore.

Here I can see that it appears correctly when it's mounted:
I8d7B.jpg


The drive is set to automatically mount at boot up in the fstab file, and this works fine:
Code: 
root@pve:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

PARTUUID=4d94eb4b-537e-b74c-b223-21085ccfc029 /mnt/SSD_240GB ext4 defaults 0 2

Here is the output of dmesg after the drive has disappeared:

Clipboard02.jpg


I initially thought it might be a power saving feature which was putting the drive to sleep. So I tried creating a small file every hour using crontab, to prevent it sleeping - but the problem persists.

I have also tried running badblocks and smartctl, both of which found no errors with the drive.

What can I do to get to the bottom of this?
 
Thanks. There are no results for warn, fail, or fatal. All the Error results are related to the Grafana/Telegraf services, which are trying to write to the drive but aren't able to:

Code: 
irectory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:11 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:21:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:21:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:21:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:21:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:21:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:51 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:51 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:51 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:21:51 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:01 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:01 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:01 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:01 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:04 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:04 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:22:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:06 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:11 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:34:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:21 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:34:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:34:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:34:24 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #5242886: comm telegraf: reading directory lblock 0
[Thu Nov 16 13:34:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:31 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
[Thu Nov 16 13:34:41 2023] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0



If I instead get rid of these using dmesg -T | grep -Eiv "grafana" | grep -Eiv "telegraf" | grep -iEe 'error', I don't have much to look at:

Code: 
[Thu Nov 16 12:50:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:51:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:52:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:53:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:54:06 2023] EXT4-fs error: 8 callbacks suppressed
[Thu Nov 16 12:54:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:55:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:56:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:57:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:58:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 12:59:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:00:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:01:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:02:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:03:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:04:06 2023] EXT4-fs error: 8 callbacks suppressed
[Thu Nov 16 13:04:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:05:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:06:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:07:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:08:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:09:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:10:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:11:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:12:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:13:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:14:06 2023] EXT4-fs error: 20 callbacks suppressed
[Thu Nov 16 13:14:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:15:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:16:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:17:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:18:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:19:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:20:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:21:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:22:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:23:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:24:06 2023] EXT4-fs error: 8 callbacks suppressed
[Thu Nov 16 13:24:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:25:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:26:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:27:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:28:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:29:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:30:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:31:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:32:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:33:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:34:06 2023] EXT4-fs error: 8 callbacks suppressed
[Thu Nov 16 13:34:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:35:21 2023] EXT4-fs error: 3 callbacks suppressed
[Thu Nov 16 13:36:21 2023] EXT4-fs error: 3 callbacks suppressed

However, I know that the drive failed before this 12:51pm, and the dmesg logs are showing before then: is there a way to return results from farther back?
 
So, the first error is not visible due to the massive noise of subsequent errors.

Please try:

Code: 
grep kernel /var/log/syslog | grep -iEe '(error|fatal|fail|warn)'  | head -100
 
Thanks both for that. @t.lamprecht I am using kernel Linux pve 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 GNU/Linux

@bbgeek17 I found that the error seemed to happen around 02:11am:

Code: 
root@pve:~# journalctl --since "02:00"
-- Journal begins at Fri 2023-01-20 17:07:18 GMT, ends at Thu 2023-11-16 14:07:51 GMT. --
Nov 16 02:00:01 pve CRON[357262]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 16 02:00:01 pve CRON[357263]: (root) CMD (/usr/bin/touch /mnt/SSD_240GB/.keepalivefile)
Nov 16 02:00:01 pve CRON[357262]: pam_unix(cron:session): session closed for user root
Nov 16 02:10:47 pve kernel: sd 7:0:0:0: [sdc] tag#5 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD
Nov 16 02:10:47 pve kernel: sd 7:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 0c 81 02 27 00 00 08 00
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Nov 16 02:11:00 pve kernel: scsi host7: uas_eh_device_reset_handler start
Nov 16 02:11:00 pve kernel: usb 2-1.6: reset high-speed USB device number 8 using ehci-pci
Nov 16 02:11:00 pve kernel: usb 2-1.6: device firmware changed
Nov 16 02:11:00 pve kernel: scsi host7: uas_eh_device_reset_handler FAILED err -19
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: Device offlined - not ready after error recovery
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: Device offlined - not ready after error recovery
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=60s
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 231415655 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=44s
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 0c 81 02 27 00 00 08 00
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 209781287 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
Nov 16 02:11:00 pve kernel: Buffer I/O error on dev sdc1, logical block 26214469, lost async page write
Nov 16 02:11:00 pve kernel: Aborting journal on device sdc1-8.
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: rejecting I/O to offline device
Nov 16 02:11:00 pve kernel: EXT4-fs error (device sdc1) in ext4_reserve_inode_write:5752: Journal has aborted
Nov 16 02:11:00 pve kernel: EXT4-fs error (device sdc1): ext4_dirty_inode:5948: inode #6554201: comm influxd: mark_inode_dirty error
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 231014399 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 16 02:11:00 pve kernel: EXT4-fs error (device sdc1) in ext4_dirty_inode:5949: Journal has aborted
Nov 16 02:11:00 pve kernel: EXT4-fs error (device sdc1) in ext4_reserve_inode_write:5752: Journal has aborted
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 231014399 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0


and here in the syslog:

Code: 
root@pve:~# cat /var/log/syslog | grep -iEe 'Nov 16 02:1'
Nov 16 02:11:00 pve kernel: [1065272.885143] scsi host7: uas_eh_device_reset_handler start
Nov 16 02:11:00 pve kernel: [1065272.969137] usb 2-1.6: reset high-speed USB device number 8 using ehci-pci
Nov 16 02:11:00 pve systemd[1]: Unmounting /mnt/SSD_240GB...
Nov 16 02:11:00 pve systemd[1]: mnt-SSD_240GB.mount: Succeeded.
Nov 16 02:11:00 pve systemd[1]: Unmounted /mnt/SSD_240GB.
Nov 16 02:11:00 pve pvestatd[942]: status update time (12.435 seconds)
Nov 16 02:11:00 pve kernel: [1065273.425229] sd 7:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Nov 16 02:11:00 pve kernel: [1065273.505120] usb 2-1.6: new high-speed USB device number 9 using ehci-pci
Nov 16 02:11:00 pve kernel: [1065273.664305] usb 2-1.6: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00
Nov 16 02:11:00 pve kernel: [1065273.664314] usb 2-1.6: New USB device strings: Mfr=2, Product=3, SerialNumber=1
Nov 16 02:11:00 pve kernel: [1065273.664318] usb 2-1.6: Product: Best USB Device
Nov 16 02:11:00 pve kernel: [1065273.664320] usb 2-1.6: Manufacturer: ULT-Best
Nov 16 02:11:00 pve kernel: [1065273.664322] usb 2-1.6: SerialNumber: 042011210DC5
Nov 16 02:11:00 pve kernel: [1065273.664857] usb-storage 2-1.6:1.0: USB Mass Storage device detected
Nov 16 02:11:00 pve kernel: [1065273.665318] usb-storage 2-1.6:1.0: Quirks match for vid 174c pid 55aa: 400000
Nov 16 02:11:00 pve kernel: [1065273.665371] scsi host6: usb-storage 2-1.6:1.0
Nov 16 02:11:05 pve kernel: [1065278.371285] EXT4-fs error: 13 callbacks suppressed
Nov 16 02:11:05 pve kernel: [1065278.371295] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6553668: comm influxd: reading directory lblock 0
Nov 16 02:11:07 pve kernel: [1065279.757938] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6554130: comm influxd: reading directory lblock 0
Nov 16 02:11:08 pve kernel: [1065280.757988] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6554130: comm influxd: reading directory lblock 0
Nov 16 02:11:09 pve kernel: [1065281.757897] EXT4-fs error (device sdc1): __ext4_find_entry:1663: inode #6554130: comm influxd: reading director

@LnxBil here is the output you requested:

Code: 
root@pve:~# grep kernel /var/log/syslog | grep -iEe '(error|fatal|fail|warn)'  | head -200
Nov 12 04:21:59 pve kernel: [727543.516777] scsi host6: uas_eh_device_reset_handler FAILED err -19
Nov 12 04:21:59 pve kernel: [727543.516788] sd 6:0:0:0: Device offlined - not ready after error recovery
Nov 12 04:21:59 pve kernel: [727543.516793] sd 6:0:0:0: Device offlined - not ready after error recovery
Nov 12 04:21:59 pve kernel: [727543.516795] sd 6:0:0:0: Device offlined - not ready after error recovery
Nov 12 04:21:59 pve kernel: [727543.516807] sd 6:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=60s
Nov 12 04:21:59 pve kernel: [727543.516819] blk_update_request: I/O error, dev sdb, sector 231107783 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.516847] sd 6:0:0:0: [sdb] tag#3 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=42s
Nov 12 04:21:59 pve kernel: [727543.516852] blk_update_request: I/O error, dev sdb, sector 200938663 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.516862] EXT4-fs warning (device sdb1): ext4_end_bio:344: I/O error 10 writing to inode 6554182 starting block 25117333)
Nov 12 04:21:59 pve kernel: [727543.516870] Buffer I/O error on device sdb1, logical block 25109141
Nov 12 04:21:59 pve kernel: [727543.516903] sd 6:0:0:0: [sdb] tag#24 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=52s
Nov 12 04:21:59 pve kernel: [727543.516919] blk_update_request: I/O error, dev sdb, sector 32603527 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Nov 12 04:21:59 pve kernel: [727543.516929] blk_update_request: I/O error, dev sdb, sector 231014399 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.516936] EXT4-fs warning (device sdb1): ext4_end_bio:344: I/O error 10 writing to inode 6554171 starting block 4075441)
Nov 12 04:21:59 pve kernel: [727543.516945] blk_update_request: I/O error, dev sdb, sector 231014399 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.516946] Buffer I/O error on device sdb1, logical block 4067249
Nov 12 04:21:59 pve kernel: [727543.516958] Buffer I/O error on dev sdb1, logical block 28868608, lost sync page write
Nov 12 04:21:59 pve kernel: [727543.516971] EXT4-fs warning (device sdb1): ext4_end_bio:344: I/O error 10 writing to inode 6554171 starting block 4075442)
Nov 12 04:21:59 pve kernel: [727543.516976] JBD2: Error -5 detected when updating journal superblock for sdb1-8.
Nov 12 04:21:59 pve kernel: [727543.517028] EXT4-fs error (device sdb1): ext4_journal_check_start:83: comm kworker/u8:3: Detected aborted journal
Nov 12 04:21:59 pve kernel: [727543.517115] blk_update_request: I/O error, dev sdb, sector 65535 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.517145] blk_update_request: I/O error, dev sdb, sector 65535 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.517167] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
Nov 12 04:21:59 pve kernel: [727543.517199] EXT4-fs (sdb1): I/O error while writing superblock
Nov 12 04:21:59 pve kernel: [727543.517224] EXT4-fs (sdb1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 6554171, error -30)
Nov 12 04:21:59 pve kernel: [727543.517245] Buffer I/O error on device sdb1, logical block 4067250
Nov 12 04:21:59 pve kernel: [727543.609431] EXT4-fs error (device sdb1): __ext4_find_entry:1663: inode #6554161: comm influxd: reading directory lblock 0
Nov 12 04:21:59 pve kernel: [727543.609485] blk_update_request: I/O error, dev sdb, sector 65535 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
Nov 12 04:21:59 pve kernel: [727543.609510] EXT4-fs (sdb1): I/O error while writing superblock
Nov 12 04:21:59 pve kernel: [727544.135371] EXT4-fs error (device sdb1): __ext4_find_entry:1663: inode #6553668: comm influxd: reading directory lblock 0
Nov 12 04:22:00 pve kernel: [727544.457378] EXT4-fs error (device sdb1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
Nov 12 04:22:00 pve kernel: [727544.457476] EXT4-fs error (device sdb1): __ext4_find_entry:1663: inode #6553603: comm grafana-server: reading directory lblock 0
 
teeeeee said:
Code: 
Nov 16 02:10:47 pve kernel: sd 7:0:0:0: [sdc] tag#5 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD
Nov 16 02:10:47 pve kernel: sd 7:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 0c 81 02 27 00 00 08 00
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Nov 16 02:11:00 pve kernel: scsi host7: uas_eh_device_reset_handler start
Nov 16 02:11:00 pve kernel: usb 2-1.6: reset high-speed USB device number 8 using ehci-pci
Nov 16 02:11:00 pve kernel: usb 2-1.6: device firmware changed
Nov 16 02:11:00 pve kernel: scsi host7: uas_eh_device_reset_handler FAILED err -19
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: Device offlined - not ready after error recovery
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: Device offlined - not ready after error recovery
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=60s
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#4 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 231415655 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=44s
Nov 16 02:11:00 pve kernel: sd 7:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 0c 81 02 27 00 00 08 00
Nov 16 02:11:00 pve kernel: blk_update_request: I/O error, dev sdc, sector 209781287 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
Nov 16 02:11:00 pve kernel: Buffer I/O error on dev sdc1, logical block 26214469, lost async page write
Nov 16 02:11:00 pve kernel: Aborting journal on device sdc1-8.
Click to expand...
So, this is it. Device offlined and then it's gone.
 
Yes, that part I could have told you without looking at the logs :)

Could you suggest how I might go about figuring out why? Thanks!
 
you can try:
- updating to newer kernel
- change usb ports
- change usb enclosure
- try things people suggest on the net, ie https://www.reddit.com/r/linux/comments/t7pxpk/are_all_of_your_usb_devices_disconnecting/

Despite USB having been around for a long time, its still a bit of a wild west, especially when you introduce a cheaply made sata/usb enclosures. It could also be the usb hub on your motherboard. This is not a PVE/hypervisor issue, which is confirmed by many reports of people having the same problem across a wide variety of Linux flavors for a while now.
There is a reason you never see USB disks used in Enterprise.

good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
@bbgeek17 Okay thanks. It's odd because it's been rock solid for over a year. And only the last few weeks I'm having this issue consistently. I feel like if it was any of the things you suggested (bad USB port, bad enclosure, kernel problem), then the problem would have arisen much sooner, and not out of the blue?
 
teeeeee said:
Okay thanks. It's odd because it's been rock solid for over a year. And only the last few weeks I'm having this issue consistently. I feel like if it was any of the things you suggested (bad USB port, bad enclosure, kernel problem), then the problem would have arisen much sooner, and not out of the blue?
Click to expand...
:) Would you say the same thing about a car not starting after it worked great for 5 years? Technology breaks and often it happens suddenly. May be there was a power spike, may be excessive humidity one day.

Keep in mind - all of the above is speculation. Your troubleshooting is limited to things you can control : cable, port, enclosure, kernel, kernel settings, new disk.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: teeeeee
teeeeee said:
uas_eh_abort_handler 0 uas-tag
Click to expand...
teeeeee said:
uas_eh_device_reset_handler start
Click to expand...
teeeeee said:
uas_eh_device_reset_handler FAILED err
Click to expand...
these are the messages you should google around for. Lots of conversations around them, nothing PVE specific.

PS
https://askubuntu.com/questions/148...ached-what-is-causing-it-uas-eh-abort-handler

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back