[SOLVED] Stuck with weird external USB drive

charleslcso · Dec 31, 2024

The external 8TB USB drive is kinda flaky...

Now I am stuck in this:

Code:

root@360g9:~# ls -l /mnt/tosh-8tb-dl360/
ls: reading directory '/mnt/tosh-8tb-dl360/': Input/output error
total 0

Code:

root@360g9:~# umount /mnt/tosh-8tb-dl360
umount: /mnt/tosh-8tb-dl360: target is busy.

Code:

root@360g9:~# eject /dev/sde
eject: /dev/sde: not found mountpoint or device with the given name

Code:

cat /var/log/syslog | grep sde
Dec 29 00:40:02 360g9 kernel: [8786957.926386] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.370784] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.371167] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.371437] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.371679] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.371912] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.372132] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.372347] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.372549] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:02 360g9 kernel: [8786958.372752] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #2: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.948793] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.949896] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.950431] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.950727] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.951034] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0
Dec 29 00:40:07 360g9 kernel: [8786962.951290] EXT4-fs error (device sde): __ext4_find_entry:1682: inode #144179202: comm rsync: reading directory lblock 0

Bash:

root@360g9:~# lvs -a
  LV              VG               Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve              twi-aotz-- 702.98g             0.00   0.26                          
  [data_tdata]    pve              Twi-ao---- 702.98g                                                  
  [data_tmeta]    pve              ewi-ao----  <7.18g                                                  
  [lvol0_pmspare] pve              ewi-------  <7.18g                                                  
  root            pve              -wi-ao----  96.00g                                                  
  swap            pve              -wi-ao----   8.00g                                                  
  vm-100-disk-0   pve1000-980Pro-1 -wi-a-----  20.00g                                                  
  vm-121-disk-0   pve1000-980Pro-1 -wi-ao----  20.00g                                                  
  vm-122-disk-0   pve1000-980Pro-1 -wi-ao----  20.00g                                                  
  vm-101-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-102-disk-0   pve2000-p31-1    -wi-ao---- 330.00g                                                  
  vm-103-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-104-disk-0   pve2000-p31-1    -wi-ao---- 400.00g                                                  
  vm-105-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-106-disk-0   pve2000-p31-1    -wi-a-----  20.00g                                                  
  vm-107-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-108-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-110-disk-0   pve2000-p31-1    -wi-a-----  20.00g                                                  
  vm-111-disk-0   pve2000-p31-1    -wi-a-----  20.00g                                                  
  vm-112-disk-0   pve2000-p31-1    -wi-ao----  20.00g                                                  
  vm-113-disk-0   pve500           -wi-a-----  20.00g                                                  
  vm-114-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-115-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-116-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-118-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-119-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-120-disk-0   pve500           -wi-ao----  20.00g                                                  
  vm-123-disk-0   pve500           -wi------- 300.00g                                                  
  vm-109-disk-0   pve900-2         -wi-ao----  20.00g                                                  
  vm-123-disk-0   pve900-2         -wi-ao---- 818.00g                                                  
  vm-124-disk-0   pve900-3         -wi-ao----  20.00g                                                  
  vm-125-disk-0   pve900-3         -wi-a-----  20.00g                                                  
root@360g9:~# vgs -a
  VG               #PV #LV #SN Attr   VSize    VFree  
  pve                1   3   0 wz--n- <837.33g  <16.00g
  pve1000-980Pro-1   1   3   0 wz--n-  931.51g  871.51g
  pve2000-p31-1      1  11   0 wz--n-   <1.82t <953.02g
  pve500             1   8   0 wz--n- <465.73g  <25.73g
  pve900-2           1   2   0 wz--n- <838.33g  336.00m
  pve900-3           1   2   0 wz--n- <838.33g <798.33g
root@360g9:~# lsblk
NAME                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                     8:0    0 838.3G  0 disk
├─sda1                                  8:1    0  1007K  0 part
├─sda2                                  8:2    0     1G  0 part
└─sda3                                  8:3    0 837.3G  0 part
  ├─pve-swap                          253:9    0     8G  0 lvm  [SWAP]
  ├─pve-root                          253:10   0    96G  0 lvm  /
  ├─pve-data_tmeta                    253:11   0   7.2G  0 lvm
  │ └─pve-data-tpool                  253:14   0   703G  0 lvm
  │   └─pve-data                      253:15   0   703G  1 lvm
  └─pve-data_tdata                    253:12   0   703G  0 lvm
    └─pve-data-tpool                  253:14   0   703G  0 lvm
      └─pve-data                      253:15   0   703G  1 lvm
sdb                                     8:16   0 465.7G  0 disk
├─pve500-vm--116--disk--0             253:0    0    20G  0 lvm
├─pve500-vm--118--disk--0             253:17   0    20G  0 lvm
├─pve500-vm--113--disk--0             253:19   0    20G  0 lvm
├─pve500-vm--115--disk--0             253:21   0    20G  0 lvm
├─pve500-vm--114--disk--0             253:23   0    20G  0 lvm
├─pve500-vm--120--disk--0             253:24   0    20G  0 lvm
└─pve500-vm--119--disk--0             253:25   0    20G  0 lvm
sdc                                     8:32   0 838.3G  0 disk
├─pve900--2-vm--109--disk--0          253:13   0    20G  0 lvm
└─pve900--2-vm--123--disk--0          253:29   0   818G  0 lvm
sdd                                     8:48   0 838.3G  0 disk
├─pve900--3-vm--124--disk--0          253:30   0    20G  0 lvm
└─pve900--3-vm--125--disk--0          253:31   0    20G  0 lvm
nvme1n1                               259:0    0 931.5G  0 disk
├─pve1000--980Pro--1-vm--100--disk--0 253:22   0    20G  0 lvm
├─pve1000--980Pro--1-vm--121--disk--0 253:27   0    20G  0 lvm
└─pve1000--980Pro--1-vm--122--disk--0 253:28   0    20G  0 lvm
nvme0n1                               259:1    0   1.8T  0 disk
├─pve2000--p31--1-vm--102--disk--0    253:1    0   330G  0 lvm
├─pve2000--p31--1-vm--103--disk--0    253:2    0    20G  0 lvm
├─pve2000--p31--1-vm--104--disk--0    253:3    0   400G  0 lvm
├─pve2000--p31--1-vm--107--disk--0    253:4    0    20G  0 lvm
├─pve2000--p31--1-vm--106--disk--0    253:5    0    20G  0 lvm
├─pve2000--p31--1-vm--108--disk--0    253:6    0    20G  0 lvm
├─pve2000--p31--1-vm--105--disk--0    253:7    0    20G  0 lvm
├─pve2000--p31--1-vm--101--disk--0    253:8    0    20G  0 lvm
├─pve2000--p31--1-vm--110--disk--0    253:16   0    20G  0 lvm
├─pve2000--p31--1-vm--111--disk--0    253:18   0    20G  0 lvm
└─pve2000--p31--1-vm--112--disk--0    253:20   0    20G  0 lvm

Code:

Filesystem           Inodes IUsed IFree IUse% Mounted on
udev                    32M   743   32M    1% /dev
tmpfs                   32M  1.6K   32M    1% /run
/dev/mapper/pve-root   6.0M  102K  6.0M    2% /
tmpfs                   32M   106   32M    1% /dev/shm
tmpfs                   32M    46   32M    1% /run/lock
/dev/fuse              256K    60  256K    1% /etc/pve

/dev/sde               233M  2.4M  231M    2% /mnt/tosh-8tb-dl360

tmpfs                  6.3M    18  6.3M    1% /run/user/0

Code:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=131955276k,nr_inodes=32988819,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26398024k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro,stripe=64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=25655)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

/dev/sde on /mnt/tosh-8tb-dl360 type ext4 (ro,relatime)

tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=26398020k,nr_inodes=6599505,mode=700,inode64)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)

I can't umount it and re-mount it...I don't want to restart PVE 7.4-19...

Please help.

waltar · Dec 31, 2024

Assuming your usb disk is a toshiba 8TB hdd ..., it shows I/O error which could be in first a filesystem error or a disk error but as it isn't shown anymore in lsblk so the disk is dead. What you can do is "umount -l /mnt/tosh-8tb-dl360" which should remove the disk from pve, see df.
Then you can unplug the usb drive and you need a new one to restore from backup what was onto. No need to reboot the pve host.

charleslcso · Dec 31, 2024

Wow, thanks alot! The -l option saved me.

Code:

root@360g9:/mnt/tosh-8tb-dl360# umount -l /mnt/tosh-8tb-dl360 
root@360g9:/mnt/tosh-8tb-dl360# ls -l
ls: reading directory '.': Input/output error
total 0
root@360g9:/mnt/tosh-8tb-dl360# cd ..
root@360g9:/mnt# umount -l /mnt/tosh-8tb-dl360 
umount: /mnt/tosh-8tb-dl360: not mounted.
root@360g9:/mnt# mount /dev/sdf /mnt/tosh-8tb-dl360/
root@360g9:/mnt# ls -l tosh-8tb-dl360/
total 464

I really don't know... the drive or the external case will die after a few days of backup jobs.... the drive seems to be back on /dev/sdf after I unplugged it and plugged it into another port.

Thanks again!

Kingneutron · Dec 31, 2024

You might need to keep the drive spinning, but long-term I would invest in a NAS

charleslcso · Jan 1, 2025

I want to write a script to check the validity of the FS and the drive is properly connected.

What approach should I use? Simply ls and lsblk?

gfngfn256 · Jan 1, 2025

It appears that the drive is using ext4, so to check the FS you should use:

Code:

fsck.ext4 /dev/sdXY

# replacing XY with the correct drive/partition

Search

Search

[SOLVED] Stuck with weird external USB drive

charleslcso

Member

waltar

Renowned Member

charleslcso

Member

Kingneutron

Renowned Member

charleslcso

Member

gfngfn256

Famous Member

We value your privacy