Update from 8.1.11 -> 8.2.2: zpool disappeared/disk empty?

PeterZ

Active Member
Jul 18, 2019
33
0
26
54
Hi

I just updated my system from 8.1.11 to 8.2.2 and whereas since v5.4, I had 'no problems at all', now 1 of my zpools has an issue.
At first the zpool disappeared and after recreating it, no data (only 393KB) is showing.

I don't understand what is going on and since I have done this process only once, long time ago, I'm puzzled.
The disk is /dev/sda so I ran
Code:
zpool create -f myzpool /dev/sda

1714005557357.png
 
Last edited:
Sorry, but you just screwed yourself. You basically reformatted the zpool and told ZFS to -f Force it.

Hope you have backups.

If you don't understand what is going on with sysadmin-level commands, ask for advice first before you lose data.
 
  • Like
Reactions: Dunuin
Hi @Kingneutron, thanks for answering even if I don't like the answer...
If you don't understand what is going on with sysadmin-level commands, ask for advice first before you lose data
You are completely right about that.
I got home and noticed that there were a lot of updates so I thought of running them when the system was less used.

TBH: I'm not that familiar with ZFS and I remembered that command when I first created the zpool.
So recreating a zpool reformats a disk?

1) What should I have done?
2) There is no way to turn this back?
3) Do you have an idea what might have happened that it disappeared in the first place?
 
> So recreating a zpool reformats a disk?

Basically, yes - you pretty much told it the equivalent of " FORMAT C: /Q /U "

1) What should I have done? ' zpool import -d /dev/disk/by-id ' # and see what comes back
If you can successfully import the pool but still don't see your data, ' zfs mount -a '

2) There is no way to turn this back? Maybe if you take the drive to a data-recovery service right away, but be prepared to spend $400 and up (probably over $1000) with no guarantee of positive results. This is why we keep drumming the 3-2-1 backup march.

3) Do you have an idea what might have happened that it disappeared in the first place? Nope. Too late to tell, sorry


If you don't have backups, you'll need to re-download / re-implement whatever environment and dataset/directory structure that was on that zpool.

Then you start planning a solid, regular, automated backup regimen to separate media (preferably a NAS that's not on the same system, but a separate hard drive will do if you can [preferably] back THAT up to something else in case of disaster.)

Yes, it sucks, and I'm not trying to be unkind here. I just have seen a LOT of these stories where someone had no backup, made an admin-level mistake, or had a power outage, or had Teh Craziness Happen, and now they have nothing to restore from. Most people don't learn the lesson unless they've either had personal data loss, or seen it happen to someone they know.

The best you can do is postulate for the worst case going forward, and vow to never let it happen again. (This includes understanding which commands are destructive - like dd, cat and rm. And it's almost never a good idea to Force something on zfs, unless you know exactly what you're doing and are positive that overriding default behavior is what you want.)

Document a DR plan, invest in backup equipment, test your restores, and rehearse a DR scenario - like restoring your main environment into a VM. Then when something does happen, you should be able to recover fairly quickly by following your established procedures.

Sincerely tho, Best of luck with your DR.
 
  • Like
Reactions: Dunuin
3) Do you have an idea what might have happened that it disappeared in the first place?
Hard to tell without the pool existing any longer.

2) There is no way to turn this back?
Only with data recovery. Either done by professionals or done yourself using software after doing a block-level backup of the disk. Both won't be cheap.
Cheapest option would be to restore your daily/hourly backups you hopefully got.

1) What should I have done?
Diagnosting the problem by looking at the logs, using commands like testdisk, zpool import, snartctl, lsblk and so on.

TBH: I'm not that familiar with ZFS
I always tell people here to learn about the storage that is used because this otherwise might sooner or later cause data-loss. And to have proper backups and a well tested disaster recovery plan. And the famous words "Raid is not a backup" and "Snapshots are not backups". As well as not to store the backups on the same disk as the VMs/LXCs.
This is a very good example why...
So I hope you got recent backups and learn from it for the future so this won't happen again.
 
Last edited:
  • Like
Reactions: Kingneutron

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!