TASK ERROR: could not activate storage, zfs error: cannot import : I/O error

vazek · Jan 21, 2023

Hi,

Suddently my pool and the datastore disapeared. Not showing in GUI or in CLI.

It was a pool of 4 3To hard drive in RaidZ1

root@pve:~# zpool status
no pools available

root@pve:~# zpool import
no pools available to import

It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root

There was something interesting when i used "zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1" but only once :

root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.

root@pve:~# zpool import -F Z-Storage
cannot import 'Z-Storage': no such pool available

root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 -d ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 -d ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1
no pools available to import

Don't know what to do now, any help apreciated.
Thanks!

apoc · Jan 21, 2023

What does a "zpool list" show you?
Sometimes I had issues with the pool name for whatever reason. But using the uid of the pool got me going...

vazek · Jan 22, 2023

Hi,
Same thing unfortunately

root@pve:~# zpool list
no pools available

apoc · Jan 22, 2023

Are your disks actually showing up?
You can. Check via CLI or ui

vazek · Jan 22, 2023

Yes;

It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root

Code:

root@pve:~# lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   2.7T  0 disk
├─sda1               8:1    0   2.7T  0 part
└─sda9               8:9    0     8M  0 part
sdb                  8:16   0   2.7T  0 disk
sdc                  8:32   0   2.7T  0 disk
├─sdc1               8:33   0   2.7T  0 part
└─sdc9               8:41   0     8M  0 part
sdd                  8:48   0   2.7T  0 disk
├─sdd1               8:49   0   2.7T  0 part
└─sdd9               8:57   0     8M  0 part
sde                  8:64   0  55.9G  0 disk
├─sde1               8:65   0  1007K  0 part
├─sde2               8:66   0   512M  0 part /boot/efi
└─sde3               8:67   0  55.4G  0 part
  ├─pve-swap       253:0    0   6.9G  0 lvm  [SWAP]
  ├─pve-root       253:1    0  13.8G  0 lvm  /
  ├─pve-data_tmeta 253:2    0     1G  0 lvm
  │ └─pve-data     253:4    0  25.9G  0 lvm
  └─pve-data_tdata 253:3    0  25.9G  0 lvm
    └─pve-data     253:4    0  25.9G  0 lvm
sdf                  8:80   0   7.3T  0 disk
├─sdf1               8:81   0   128M  0 part
└─sdf2               8:82   0   7.3T  0 part
sdg                  8:96   0 931.5G  0 disk
├─sdg1               8:97   0   499M  0 part
├─sdg2               8:98   0    99M  0 part
├─sdg3               8:99   0    16M  0 part
├─sdg4               8:100  0 930.4G  0 part
└─sdg5               8:101  0   550M  0 part
sdh                  8:112  0 465.8G  0 disk
├─sdh1               8:113  0   128M  0 part
├─sdh2               8:114  0   499M  0 part
├─sdh3               8:115  0    99M  0 part
└─sdh4               8:116  0 465.1G  0 part

vazek · Jan 24, 2023

Anyone?

leesteken · Jan 24, 2023

vazek said:
root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.

Looks to me that your experienced one drive failture (which it can handle) and some additional disk corruption or read/write failures, which it cannot handle because there is no copy/parity anymore.
Did you try using -F instead of -f?

vazek · Jan 24, 2023

Yes i tried both, with disk name and zpool name.

But i got the message above only once, the pool is now never seem by any command, this is why it's hard to do anything.

vazek · Jan 25, 2023

somebody once told me the world is gonna roll me

leesteken · Jan 25, 2023

Unless you find a ZFS specialist, your pool is probably unrecoverable because it encountered more errors than its one disk redundancy could handle.
I advise double redundancy, so that your data is not instantly at risk when the first one fails, and regular (off-site) backups.

vazek · Jan 25, 2023

I don't think that is the case, because i have smart alerts.
I think there is one disk that failed for sure, but maybe it's the perc controller (in hba) i'm using or the cables, so far i haven't tried that.
In fact i just booted to a live debian and my pool is finnaly seen there :


user@debian:~$ sudo zpool import
   pool: Z-Storage
     id: 7850785934179015590
  state: DEGRADED
status: The pool was last accessed by another system.
 action: The pool can be imported despite missing or damaged devices.  The
    fault tolerance of the pool may be compromised if imported.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

    Z-Storage                   DEGRADED
      raidz1-0                  DEGRADED
        wwn-0x50014ee2ba381480  ONLINE
        wwn-0x50014ee2651b740f  ONLINE
        10679196683664851383    UNAVAIL
        wwn-0x50014ee059ab5348  ONLINE

leesteken · Jan 25, 2023

vazek said:
In fact i just booted to a live debian and my pool is finnaly seen there :

Lucky you! I mean; good thinking to try an see if another system can access the pool! Quickly replace the missing drive and resilver it. I have no idea why it won't work in the original system.

alexskysilk · Jan 25, 2023

just curious- is /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 the drive that failed?

vazek · Jan 31, 2023

just curious- is /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 the drive that failed?

No, it was ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9. It was not even a year old! It's a WD30EFZX model, all the others are WD30EFRX.
I also replaced the ssd with the proxmox install, as it was a no-name one from 2016. Maybe it was failling too and because of that i couldn't see my pool?
I reimported my pool and resilvered, works fine now.
Thanks for the help!

Search

Search

TASK ERROR: could not activate storage, zfs error: cannot import : I/O error

vazek

New Member

apoc

Famous Member

vazek

New Member

apoc

Famous Member

vazek

New Member

vazek

New Member

leesteken

Distinguished Member

vazek

New Member

vazek

New Member

leesteken

Distinguished Member

vazek

New Member

leesteken

Distinguished Member

alexskysilk

Distinguished Member

vazek

New Member