TASK ERROR: could not activate storage, zfs error: cannot import : I/O error

vazek

New Member
Jul 9, 2022
8
1
3
Hi,

Suddently my pool and the datastore disapeared. Not showing in GUI or in CLI.

It was a pool of 4 3To hard drive in RaidZ1

root@pve:~# zpool status
no pools available
root@pve:~# zpool import
no pools available to import
It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root


There was something interesting when i used "zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1" but only once :

root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.
root@pve:~# zpool import -F Z-Storage
cannot import 'Z-Storage': no such pool available
root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 -d ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 -d ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1
no pools available to import


Don't know what to do now, any help apreciated.
Thanks!
 
Last edited:
What does a "zpool list" show you?
Sometimes I had issues with the pool name for whatever reason. But using the uid of the pool got me going...
 
Yes;

It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root
Code:
root@pve:~# lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   2.7T  0 disk
├─sda1               8:1    0   2.7T  0 part
└─sda9               8:9    0     8M  0 part
sdb                  8:16   0   2.7T  0 disk
sdc                  8:32   0   2.7T  0 disk
├─sdc1               8:33   0   2.7T  0 part
└─sdc9               8:41   0     8M  0 part
sdd                  8:48   0   2.7T  0 disk
├─sdd1               8:49   0   2.7T  0 part
└─sdd9               8:57   0     8M  0 part
sde                  8:64   0  55.9G  0 disk
├─sde1               8:65   0  1007K  0 part
├─sde2               8:66   0   512M  0 part /boot/efi
└─sde3               8:67   0  55.4G  0 part
  ├─pve-swap       253:0    0   6.9G  0 lvm  [SWAP]
  ├─pve-root       253:1    0  13.8G  0 lvm  /
  ├─pve-data_tmeta 253:2    0     1G  0 lvm
  │ └─pve-data     253:4    0  25.9G  0 lvm
  └─pve-data_tdata 253:3    0  25.9G  0 lvm
    └─pve-data     253:4    0  25.9G  0 lvm
sdf                  8:80   0   7.3T  0 disk
├─sdf1               8:81   0   128M  0 part
└─sdf2               8:82   0   7.3T  0 part
sdg                  8:96   0 931.5G  0 disk
├─sdg1               8:97   0   499M  0 part
├─sdg2               8:98   0    99M  0 part
├─sdg3               8:99   0    16M  0 part
├─sdg4               8:100  0 930.4G  0 part
└─sdg5               8:101  0   550M  0 part
sdh                  8:112  0 465.8G  0 disk
├─sdh1               8:113  0   128M  0 part
├─sdh2               8:114  0   499M  0 part
├─sdh3               8:115  0    99M  0 part
└─sdh4               8:116  0 465.1G  0 part
 
Last edited:
root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.
Looks to me that your experienced one drive failture (which it can handle) and some additional disk corruption or read/write failures, which it cannot handle because there is no copy/parity anymore.
Did you try using -F instead of -f?
 
Yes i tried both, with disk name and zpool name.

But i got the message above only once, the pool is now never seem by any command, this is why it's hard to do anything.
 
Unless you find a ZFS specialist, your pool is probably unrecoverable because it encountered more errors than its one disk redundancy could handle.
I advise double redundancy, so that your data is not instantly at risk when the first one fails, and regular (off-site) backups.
 
I don't think that is the case, because i have smart alerts.
I think there is one disk that failed for sure, but maybe it's the perc controller (in hba) i'm using or the cables, so far i haven't tried that.
In fact i just booted to a live debian and my pool is finnaly seen there :
user@debian:~$ sudo zpool import pool: Z-Storage id: 7850785934179015590 state: DEGRADED status: The pool was last accessed by another system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY config: Z-Storage DEGRADED raidz1-0 DEGRADED wwn-0x50014ee2ba381480 ONLINE wwn-0x50014ee2651b740f ONLINE 10679196683664851383 UNAVAIL wwn-0x50014ee059ab5348 ONLINE
 
Last edited:
  • Like
Reactions: leesteken
In fact i just booted to a live debian and my pool is finnaly seen there :
Lucky you! I mean; good thinking to try an see if another system can access the pool! Quickly replace the missing drive and resilver it. I have no idea why it won't work in the original system.
 
Last edited:
just curious- is /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 the drive that failed?

No, it was ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9. It was not even a year old! It's a WD30EFZX model, all the others are WD30EFRX.
I also replaced the ssd with the proxmox install, as it was a no-name one from 2016. Maybe it was failling too and because of that i couldn't see my pool?
I reimported my pool and resilvered, works fine now.
Thanks for the help!