Lost all data on ZFS RAID10

[jhr]

Hello,

I just lost all data from my test lab server. It's Supermicro X12STH-LN4F. Proxmox VE was installed to consumer NVME SSD Patriot P300. ZFS RAID1 is placed on 4x HGST HUS722T2TAL. Slow, but functional.

My server freeze in the night. I was not able to reach it, ping not working too. After hard reset, it hangs at the DXE-AHCI Initialization stage forever. I tried to remove all disks and it boots again. After that I noticed, that NVME SSD is completlty dead, OK, I replaced it with another one and did a fresh install of Proxmox VE.
After that I imported my previous zpool named zfs with zpool import -f zfs, which runs fine, but whole area is empty. zfs list shows none of my previously created partitions.
Using zpool history -il I can view history of that pool, but how can I restore my data ? Is it normal behavior ?

This is only test server, so there are nothing important to me, but what if the same situations happend on production environment ?

Thanks

 

[waltar]

If your test server would be in production yet what would you expect to happen in same situation ?

 

[jhr]

Well, I can accept a risk of NVME damage - Proxmox OS, but I don't accept loosing of all data. I don't know why this happend.

 

[waltar]

Yes, well, in one way zfs is the savest filesystem in theory but at the same times it's the most sensitive filesystem in reality in special to any power outage also so a usv with server-power-down-software should be mandatory ... but in your case, mmh, that wouldn't help there either.
I'm impressed by the features zfs has but in useability I'm mostly disappointed about. But nevertheless it's in everyone own decision and experience while mostly had good one until any day coming the unexpected.

 

[jhr]

Well, this is the first serious problem with ZFS I have after years of using it.

 

[waltar]

Just a 300TB zpool fileserver about 50% used had a power outage little over 1 month before, few colleagues tried for few hours to reimport unsuccessfull and after all restored for a few days from luckily existing DR-server with send/receive. Puh, whole development department wasn't able to work for that time and I was very very happy not to be responsible for that at all.

 

[floh8]

Therefore it's fundamental to have a working backup. Always, independent which filesystem u use.

 

[jhr]

@waltar
I understand, but my server had no power outage at all.

@floh8
I have working backups on all production environment, but I make one backup per day, so there will be always some data loosing.

 

[JensF]

Do you get any output executing zfs list?
I guess your VM disks are still there, but you lost your configs (which were stored on the broken NVME).

 

[waltar]

The most I'm searching for is how to go backward in zfs transactions to get a corrupted data zpool imported again but there's no crashed pool available to play with also as those are needed back immedently ... a problem which looks as couldn't be solved for a production pool unluckily.

 

[jhr]

zfs list was pretty empty, USED 300MB, the rest were AVAILABLE

 

[jhr]

Do you get any output executing zfs list?
I guess your VM disks are still there, but you lost your configs (which were stored on the broken NVME).

I am not talking about VM/CT configs, I am looking for non existent partitions

 

[waltar]

non existent partitions

?? Mostly zfs disk labels were corruped but still there but none existing partitions never heard before ... hmm, "parted -l" say empty disks ??

 

[showiproute]

Just as a side note: Proxmox will store all relevat VM information including config on the main OS disk.
Keep that in mind.
You won't loose any data from the VM itself if it sits on healthy drive but you would need to rebuild the VM by hand

 

[jhr]

?? Mostly zfs disk labels were corruped but still there but none existing partitions never heard before ... hmm, "parted -l" say empty disks ??

No, on all disks are all partitions related to ZFS RAID10, so I could zpool import, but without any ZFS subvols

 

[jhr]

Just as a side note: Proxmox will store all relevat VM information including config on the main OS disk.
Keep that in mind.
You won't loose any data from the VM itself if it sits on healthy drive but you would need to rebuild the VM by hand

I know, I relay on fact that all configs are included in backup on PBS, so I can take all configs from PBS. I can restore all VMs too, but I have backups 20hours old, so I loose data and want to know what happend and how to deal with this situation in the future

 

[waltar]

When "zpool import ..." works then "zfs list" will/should show your vm's as zvols and that doesn't work yet while the import did ??

 

[esi_y]

Just as a side note: Proxmox will store all relevat VM information including config on the main OS disk.
Keep that in mind.
You won't loose any data from the VM itself if it sits on healthy drive but you would need to rebuild the VM by hand

You should be backing that up as well.

https://forum.proxmox.com/threads/backup-cluster-config-pmxcfs-etc-pve.154569/

 

[jhr]

When "zpool import ..." works then "zfs list" will/should show your vm's as zvols and that doesn't work yet while the import did ??

zpool status shows no pool. So I import and than I can see my imported pool, but zfs list not showing my zvols

 

[esi_y]

I know, I relay on fact that all configs are included in backup on PBS, so I can take all configs from PBS. I can restore all VMs too, but I have backups 20hours old, so I loose data and want to know what happend and how to deal with this situation in the future

Is there any reason using ZFS to begin with? I am always wondering when that filesystem was being devised, how they were taking into account virtualisation uses and SSDs, etc.