ZFS Pool missing after power loss

[sublimnl]

hi all, i had a power outage this morning and one of my zfs pools did not come back online. i am seeing errors like the following:

Dec 24 09:41:08 p330 zpool[636]: cannot import 'p330-nvme02': no such pool available
Dec 24 09:41:46 p330 pvestatd[1428]: zfs error: cannot open 'p330-nvme02': no such pool
Dec 24 09:41:46 p330 pvestatd[1428]: could not activate storage 'p330-nvme02', zfs error: cannot import 'p330-nvme02': no such pool available

I had a look in the BIOS and the NVME is still there and i can also see the following...

# ls -lah /dev/nvme*
crw------- 1 root root 241, 0 Dec 24 10:04 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Dec 24 10:04 /dev/nvme0n1
brw-rw---- 1 root disk 259, 2 Dec 24 10:04 /dev/nvme0n1p1
brw-rw---- 1 root disk 259, 3 Dec 24 10:04 /dev/nvme0n1p9
crw------- 1 root root 241, 1 Dec 24 10:04 /dev/nvme1
brw-rw---- 1 root disk 259, 1 Dec 24 10:04 /dev/nvme1n1

# lsblk | grep nvme
nvme0n1                      259:0    0   3.7T  0 disk
├─nvme0n1p1                  259:2    0   3.7T  0 part
└─nvme0n1p9                  259:3    0     8M  0 part
nvme1n1                      259:1    0   3.7T  0 disk

fdisk also doesn't see the partitions:

Disk /dev/nvme0n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: Lexar SSD NM790 4TB                   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9F7CC08A-EAB3-7643-B029-5577A8580D67

Device              Start        End    Sectors  Size Type
/dev/nvme0n1p1       2048 8001556479 8001554432  3.7T Solaris /usr & Apple ZFS
/dev/nvme0n1p9 8001556480 8001572863      16384    8M Solaris reserved 1


Disk /dev/nvme1n1: 3.73 TiB, 4096805658624 bytes, 8001573552 sectors
Disk model: Lexar SSD NM790 4TB                   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

can anyone provide some guidance on what, if anything, I can do to get nvme1n1 back up and running? thanks!

 

[waltar]

Unfortunately I thought there's nothing anymore to get back when even the zfs partitions (1+9) were not there. You should get a new nvme and a restore of your backup.

 

[Kingneutron]

Situations like this are why it's important to have a UPS, and clean shutdowns.

 

[sublimnl]

Yep. I opened an RMA with Lexar which they have been very responsive on. They've confirmed it is bad and are replacing under warranty. Thanks guys.