Problem with Storage HDD disks - Input/output errors

[marcor78]

Hello,
I really can't figure out what's going on with my storage....
Few days ago I started experiencing some input/output errors on one of my storage disk (HDD) containing some of my images (vm started crushing obviously).
I thought it was a hardware failure (corrupted disk), so I replaced with a new one (Seagate Expansion 2TB), restored images from backup and so on.
Now I see erratically same behaviour with the new disk: input/output errors, disk unmounted and with question mark, vm crushing again..

After a long search I found that the Seagate disk is somehow blacklisted because of its firmware, therefore it didn't even show S.M.A.R.T. values (ref. https://www.smartmontools.org/wiki/SAT-with-UAS-Linux#Workarounds). Following the workaround, I disabled UAS for the device, and S.M.A.R.T. values finally appeared.

Nonetheless, after a couple of hours, new input/output errors and all the disks unmounted again (even other WD disks that always performed very good).

I really don't know what else to try... Thanks for anyone could help with this issue!

here's my pveversion:

# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-4
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-5
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

and this is my /etc/fstab on the pve:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=1655-01C8 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
LABEL=storage /home/Storage ext4 defaults 0 2
LABEL=storage2 /home/Storage2 ext4 defaults 0 2
LABEL=images /home/Images ext4 defaults 0 2
LABEL=backup /home/Backup ext4 defaults 0 2
cgroup /sys/fs/cgroup cgroup defaults

finally, this is my lsblk with all the disks mounted (the 2 seagate disks are sde and sdh):

lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                          7:0    0     8G  0 loop
loop1                          7:1    0     5G  0 loop
loop2                          7:2    0     8G  0 loop
loop3                          7:3    0     8G  0 loop
sda                            8:0    0 111.8G  0 disk
├─sda1                         8:1    0  1007K  0 part
├─sda2                         8:2    0   512M  0 part /boot/efi
└─sda3                         8:3    0 111.3G  0 part
  ├─pve-swap                 253:0    0     7G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0  27.8G  0 lvm  /
  ├─pve-data_tmeta           253:2    0     1G  0 lvm
  │ └─pve-data-tpool         253:4    0  60.7G  0 lvm
  │   ├─pve-data             253:5    0  60.7G  1 lvm
  │   ├─pve-vm--211--disk--0 253:6    0     8G  0 lvm
  │   ├─pve-vm--214--disk--0 253:7    0    10G  0 lvm
  │   ├─pve-vm--200--disk--0 253:8    0     4M  0 lvm
  │   └─pve-vm--200--disk--1 253:9    0    32G  0 lvm
  └─pve-data_tdata           253:3    0  60.7G  0 lvm
    └─pve-data-tpool         253:4    0  60.7G  0 lvm
      ├─pve-data             253:5    0  60.7G  1 lvm
      ├─pve-vm--211--disk--0 253:6    0     8G  0 lvm
      ├─pve-vm--214--disk--0 253:7    0    10G  0 lvm
      ├─pve-vm--200--disk--0 253:8    0     4M  0 lvm
      └─pve-vm--200--disk--1 253:9    0    32G  0 lvm
sde                            8:64   0   1.8T  0 disk
└─sde1                         8:65   0   1.8T  0 part /home/Storage2
sdg                            8:96   0 465.7G  0 disk
└─sdg1                         8:97   0 465.7G  0 part /home/Storage
sdh                            8:112  0   1.8T  0 disk
├─sdh1                         8:113  0   500G  0 part /home/Images
└─sdh2                         8:114  0   1.3T  0 part /home/Backup

 

[leesteken]

Maybe it's the SATA controller or cables? Or maybe your RAM is failing and introduces errors (this tends to get worse over time): try running a memtest86.

 

[LnxBil]

Maybe it's the SATA controller or cables? Or maybe your RAM is failing and introduces errors (this tends to get worse over time): try running a memtest86.

or maybe the power supply .. or even too much radiation (hopefully not).

 

[marcor78]

I'm trying to test disks one by one, excluding them and trying to isolate possible disk failures... I'm still testing, but it seems that keeping the seagate disk mounted, the system tends to generate input/output errors also on the non-seagate disk (which from smartctl checks doesn't show any failure). Now I'm trying with only the non-seagate disk mounted (seagate unmounted) and the system hasn't crushed anymore since a few hours.
So in the end it's probably just a problem with the seagate disk. I couldn't imagine this, because the disk is brand new... but... can't be anything else I'm afraid, just wasted money...