[SOLVED] ZFS pool failure after power outage

fa1

Member
Jun 4, 2021
3
1
8
Hi, I would like to ask for advice. We've had a UPS failure in the server room and one of our boxes couldn't import zfs pool named spool after the box rebooted. The error message was:

# zpool import spool
cannot import 'spool': I/O error
Destroy and re-create the pool from
a backup source.
# zpool import -F spool
cannot import 'spool': I/O error
Destroy and re-create the pool from
a backup source.
# zpool import -fF spool
cannot import 'spool': I/O error
Destroy and re-create the pool from
a backup source.

Running zpool import -fFX at least started and it now runs more than 60 hours on this 8x4TB striped, mirrored (raid10) pool . The documentation states the extreme rewind import (import -X) performs a full traversal of all blocks in the pool for verification. I'm also monitoring the disk data reads with iostat, and it shows 11.5TB total reads per disk at the moment. Is there any method to discover about the status for the recovery process?

Code:
# iostat -m /dev/sd?
Linux 4.15.18-30-pve ()      06/04/2021      _x86_64_        (40 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.05    0.00    2.80    0.21    0.00   96.93

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda             526.84        54.45         0.00   11955208          4
sdc             518.58        53.80         0.00   11811866         14
sdd             517.60        53.79         0.00   11811047         11
sde             514.48        53.67         0.00   11784840          9
sdf             516.71        53.35         0.00   11713605          4
sdg             524.74        54.97         0.00   12069389          4
sdh             515.77        53.35         0.00   11713263          4
sdi             511.17        53.66         0.00   11783155         13
sdj             518.26        54.43         0.00   11951749          4
sdk             520.75        54.44         0.00   11953252          4
sdl             527.27        54.97         0.00   12070367          4
sdb             525.12        54.44         0.00   11954299         16

Any help appriciated.

Thanks in advance.
 
After 70 hours and 12.5tb/disk data read the zpool import -fFX spool command restored everything (15TB) back. The pool is ONLINE and coherent. The per disk read speed were fluctuating between 1-130mb/s during the restore averaging arount 50mb/s.
 
70 hours ... I think a restore from backup would be much faster. But yes if you didnt backup the data prior...

All those drives maybe time for a UPS?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!