Cannot boot pve... cannot import 'rpool', cache problem?

[ReenigneArcher]

After safely shutting down my PVE server during a power outage, I am getting the following error when trying to boot it up again. (I typed this out since I can't copy and paste from the server, so it's not 100% accurate, but close enough)

Loading Linux 5.15.74-1-pve
Loading initial ramdisk ...
[13.578642] mpt2sas_cm0: overriding NVDATA EEDPTagMode setting

Command /sbin/zpool import -c /etc/zfs/zpool.cache -N 'rpool'
Message: cannot import 'rpool': I/O error
cannot import 'rpool': I/O error
Destroy and re-create the pool from
a backup source.
cachefile import failed, retrying
Destroy and re-create the pool from
a backup source.
Error: 1

Failed to import pool 'rpool'
Manually import the pool and exit.

I then get put into BusyBox v1.30.1 with a command line prefix of (initramfs)

I tried adding a rootdelay to the grub command by pressing `e` on the grub menu and adding `rootdelay=10` before the `quiet` then pressing Ctrl+x. I also tried in recovery mode, but the issue is the same. I also tried `zpool import -N rpool -f` but got the same error.

My boot drives are 2 nvme SSDs mirrored. How can I recover? Any assistance would be greatly appreciated.

 

[ReenigneArcher]

I was slightly mistaken. My boot drives are 2 sata ssds, mirrored.

I did a strait `zpool import` and I can see a little more info.

pool: rpool
id: blahblahblah
state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
config:
rpool FAULTED corrupted data
mirror-0 ONLINE
ata... ONLINE
ata... ONLINE

Is there anyway to repair the pool? Hopefully there is, otherwise what is the point of a mirror?

If not, can I reinstall proxmox and keep my VMs? I put all my virtual disks on other pools/drives.

 

[Kingneutron]

https://github.com/nchevsky/systemrescue-zfs

Unfortunately, RAID/mirror is not a backup. Redundancy is useful for uptime, but it doesn't help if both disks are affected.

https://github.com/kneutron/ansitest/tree/master/proxmox

Setup and run the bkpcrit script, backup to other media / NAS (which can be as simple as your win10 desktop with a Samba shared drive and sufficient free disk space, or to another Linux instance with sshfs.)


This should hopefully at least get you a copy of /etc if you can manage to mount the pool readonly.

After you reinstall, restoring your critical files should re-populate the VMs in the GUI, as long as the storage is defined the same way beforehand.

When you get back up and running, setup some kind of regular backup (bkpcrit nightly in cron is a good start) - and there is a script in that repo that allows you to attach a 3rd mirror disk temporarily to create a DR boot disk - although keeping it updated is up to you.

The 3rd disk/mirror can be a spinner HD if you want, it doesn't necessarily have to be SSD media since it will be saved for emergency situations.


Further reading, when you're in a better state of mind:

https://serverfault.com/questions/2888/why-is-raid-not-a-backup

 

[ReenigneArcher]

Thanks for the response and yea, I know redundancy is not a backup... But it's kind of screwy that data gets corrupted when doing a safe shutdown too.

I ended up "fixing" this by running the following.

dryrun:

zpool import -f -FXn rpool

Which indicated that I would lose about 16 days of transactions. I went for it since I hadn't made any config or VM changes in that time frame anyway.

zpool import -f -FX rpool

The zfs error messages were also not that clear since it indicated the only option was to destroy the rpool and re-create it, which was not actually the only option.