TASK ERROR: could not activate storage, zfs error: cannot import : I/O error

vazek

New Member
Jul 9, 2022
8
1
3
Hi,

Suddently my pool and the datastore disapeared. Not showing in GUI or in CLI.

It was a pool of 4 3To hard drive in RaidZ1

root@pve:~# zpool status
no pools available
root@pve:~# zpool import
no pools available to import
It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root


There was something interesting when i used "zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1" but only once :

root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.
root@pve:~# zpool import -F Z-Storage
cannot import 'Z-Storage': no such pool available
root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 -d ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 -d ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1
no pools available to import


Don't know what to do now, any help apreciated.
Thanks!
 
Last edited:
What does a "zpool list" show you?
Sometimes I had issues with the pool name for whatever reason. But using the uid of the pool got me going...
 
Are your disks actually showing up?
You can. Check via CLI or ui
 
Yes;

It looks like one drive failed (no more partition on it) But the pool should only be degrated?

https://zupimages.net/viewer.php?id=23/03/wh39.png

root@pve:~# ls /dev/disk/by-id/
ata-CT1000MX500SSD1_1910E1EEA792 dm-name-pve-swap
ata-CT1000MX500SSD1_1910E1EEA792-part1 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4N2xe9bcWSqz83KdvKxwyWN0SyCHws5FVq
ata-CT1000MX500SSD1_1910E1EEA792-part2 dm-uuid-LVM-APkR5L7IRXy2EALHX4ZtOHpXx7i48d4NAZRxdrKcwx3rZC9cv9QwAB3NFdJN9IbI
ata-CT1000MX500SSD1_1910E1EEA792-part3 lvm-pv-uuid-gFdYNU-GuDd-M1qr-sxpi-qRt5-lAGK-jdNudB
ata-CT1000MX500SSD1_1910E1EEA792-part4 wwn-0x5000c500a43bddbc
ata-CT1000MX500SSD1_1910E1EEA792-part5 wwn-0x5000c500a43bddbc-part1
ata-LDLC_05022217A0127 wwn-0x5000c500a43bddbc-part2
ata-LDLC_05022217A0127-part1 wwn-0x50014ee059ab5348
ata-LDLC_05022217A0127-part2 wwn-0x50014ee059ab5348-part1
ata-LDLC_05022217A0127-part3 wwn-0x50014ee059ab5348-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H wwn-0x50014ee2651b740f
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part1 wwn-0x50014ee2651b740f-part1
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part2 wwn-0x50014ee2651b740f-part9
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part3 wwn-0x50014ee2ba381480
ata-Samsung_SSD_860_EVO_500GB_S3Z2NB2KA00187H-part4 wwn-0x50014ee2ba381480-part1
ata-ST8000VN0022-2EL112_ZA18QA85 wwn-0x50014ee2ba381480-part9
ata-ST8000VN0022-2EL112_ZA18QA85-part1 wwn-0x50014ee2bf24afd9
ata-ST8000VN0022-2EL112_ZA18QA85-part2 wwn-0x5002538e408d218f
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 wwn-0x5002538e408d218f-part1
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1 wwn-0x5002538e408d218f-part2
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part9 wwn-0x5002538e408d218f-part3
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5 wwn-0x5002538e408d218f-part4
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part1 wwn-0x500a0751e1eea792
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5FS2XJ5-part9 wwn-0x500a0751e1eea792-part1
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX wwn-0x500a0751e1eea792-part2
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part1 wwn-0x500a0751e1eea792-part3
ata-WDC_WD30EFRX-68N32N0_WD-WCC7K2ZK9ADX-part9 wwn-0x500a0751e1eea792-part4
ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9 wwn-0x500a0751e1eea792-part5
dm-name-pve-root
Code:
root@pve:~# lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   2.7T  0 disk
├─sda1               8:1    0   2.7T  0 part
└─sda9               8:9    0     8M  0 part
sdb                  8:16   0   2.7T  0 disk
sdc                  8:32   0   2.7T  0 disk
├─sdc1               8:33   0   2.7T  0 part
└─sdc9               8:41   0     8M  0 part
sdd                  8:48   0   2.7T  0 disk
├─sdd1               8:49   0   2.7T  0 part
└─sdd9               8:57   0     8M  0 part
sde                  8:64   0  55.9G  0 disk
├─sde1               8:65   0  1007K  0 part
├─sde2               8:66   0   512M  0 part /boot/efi
└─sde3               8:67   0  55.4G  0 part
  ├─pve-swap       253:0    0   6.9G  0 lvm  [SWAP]
  ├─pve-root       253:1    0  13.8G  0 lvm  /
  ├─pve-data_tmeta 253:2    0     1G  0 lvm
  │ └─pve-data     253:4    0  25.9G  0 lvm
  └─pve-data_tdata 253:3    0  25.9G  0 lvm
    └─pve-data     253:4    0  25.9G  0 lvm
sdf                  8:80   0   7.3T  0 disk
├─sdf1               8:81   0   128M  0 part
└─sdf2               8:82   0   7.3T  0 part
sdg                  8:96   0 931.5G  0 disk
├─sdg1               8:97   0   499M  0 part
├─sdg2               8:98   0    99M  0 part
├─sdg3               8:99   0    16M  0 part
├─sdg4               8:100  0 930.4G  0 part
└─sdg5               8:101  0   550M  0 part
sdh                  8:112  0 465.8G  0 disk
├─sdh1               8:113  0   128M  0 part
├─sdh2               8:114  0   499M  0 part
├─sdh3               8:115  0    99M  0 part
└─sdh4               8:116  0 465.1G  0 part
 
Last edited:
root@pve:~# zpool import -a -f -d /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7-part1
cannot import 'Z-Storage': I/O error
Recovery is possible, but will result in some data loss.
Returning the pool to its state as of Fri 20 Jan 2023 05:58:50 AM CET
should correct the problem. Approximately 28 seconds of data
must be discarded, irreversibly. Recovery can be attempted
by executing 'zpool import -F Z-Storage'. A scrub of the pool
is strongly recommended after recovery.
Looks to me that your experienced one drive failture (which it can handle) and some additional disk corruption or read/write failures, which it cannot handle because there is no copy/parity anymore.
Did you try using -F instead of -f?
 
Yes i tried both, with disk name and zpool name.

But i got the message above only once, the pool is now never seem by any command, this is why it's hard to do anything.
 
Unless you find a ZFS specialist, your pool is probably unrecoverable because it encountered more errors than its one disk redundancy could handle.
I advise double redundancy, so that your data is not instantly at risk when the first one fails, and regular (off-site) backups.
 
I don't think that is the case, because i have smart alerts.
I think there is one disk that failed for sure, but maybe it's the perc controller (in hba) i'm using or the cables, so far i haven't tried that.
In fact i just booted to a live debian and my pool is finnaly seen there :
user@debian:~$ sudo zpool import pool: Z-Storage id: 7850785934179015590 state: DEGRADED status: The pool was last accessed by another system. action: The pool can be imported despite missing or damaged devices. The fault tolerance of the pool may be compromised if imported. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY config: Z-Storage DEGRADED raidz1-0 DEGRADED wwn-0x50014ee2ba381480 ONLINE wwn-0x50014ee2651b740f ONLINE 10679196683664851383 UNAVAIL wwn-0x50014ee059ab5348 ONLINE
 
Last edited:
  • Like
Reactions: leesteken
In fact i just booted to a live debian and my pool is finnaly seen there :
Lucky you! I mean; good thinking to try an see if another system can access the pool! Quickly replace the missing drive and resilver it. I have no idea why it won't work in the original system.
 
Last edited:
just curious- is /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0SSFRA7 the drive that failed?

No, it was ata-WDC_WD30EFZX-68AWUN0_WD-WX12D81FZCR9. It was not even a year old! It's a WD30EFZX model, all the others are WD30EFRX.
I also replaced the ssd with the proxmox install, as it was a no-name one from 2016. Maybe it was failling too and because of that i couldn't see my pool?
I reimported my pool and resilvered, works fine now.
Thanks for the help!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!