How to recover guests on local ZFS after host broke?

[budy]

I am still exploring the two most interesting storage solultions for my home/lab setup, which are Ceph and ZFS. I know I can make Ceph be fine with just 3 or 4 OSDs running on the same node, if the crush_map gets tweaked accordingly. However, I am already a ZFS old-fart and I also looked into ZFS as local storage on a 2x 2-vdev mirror.

What bothers me is, that there seems no easy way to recover from a host failiure, when you'd need to setup PVE again ending up with a blank PVE and your guests gone. My basic question is: how can I salvage my quests, which are still all fine on my zpool? I haven't found a way to get any of those raw disks be used again.

 

[apoc]

My approach would be to not use the Proxmox installer (iso)

While this is very comfortable it was limited in some ways (back when I tried it partitioning wasn't customizable).
I have been in the situation as well (trying to boot from a very small USB-device) so I ended up in installing Debian as a "custom image" and later on adding Proxmox on top of it.
Worked like a charm for me. Even the upgrade from PVE5 to PVE6 worked without any issues for me (no complex configuration though).

Basically I have used this one:
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch

But there is also a new version available of it:
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster

If you get the idea also to use a USB-Stick: don't make my mistake in going down the path.
Get yourself a small SSD and boot from it. Even when attached to USB these drives do a far better job in wear-leveling.

Also you can make an image of that small boot disk via DD and restore from that if necessary.
Additionally you can copy off (I would recommend in doing so) the VM config files which are stored in /etc/pve/quemu-server

 

[budy]

Also you can make an image of that small boot disk via DD and restore from that if necessary.
Additionally you can copy off (I would recommend in doing so) the VM config files which are stored in /etc/pve/quemu-server

Well yes, that will be an option, but what if you don't have that and are only left with the ZPOOL holding your raw disk images? Isn't there a way to get them active again? You can't create a new guest with the same ID as the disks, that's for sure…

 

[apoc]

the easiest way is to backup your VM configuration.
This will keep everything "in order" in terms of VMs and their disks. The VM-IDs keep the same and they refer to the ZFS-Pool / ZVOL.
What you would need to do on a reinstall is: Create the ZFS storage with the same name. And all references will find their target.

If you build new VMs, they likely will get different IDs (due to the way Proxmox handles VM-creation). You can't change their IDs from the UI, but can rename the .conf files in /etc/pve/qemu-server. The UI will re-read the changes. In my case I have manually modified severall VMs to "group them by ID".
You also can rename the ZVOLs by using "zfs rename <oldname> <newname>" so the ZVOL are matching the expected name of PVE.

In the very end it is manual, cumbersome and error prone.
Therefore I'd stick with my recommendation to make an image of the Proxmox-Installation. It is just the simplest way of recovery.
It has saved me multiple times, so I guess it will work for you as well.

My 8GB OS image is zipped around 4GB in size - happy to "waste" that storage for an easy recovery of my virtualization host.

 

[budy]

I don't object to that! As long as you have managed to backup that data in time, this is a viable solution. I was wondering what the options were, if one hadn't done that.

…not that I intend on letting that slip, of course. And yes, I will also create a disk image of my install disk…

Thanks!

 

[apoc]

A better solution imho would be to backup the conf-files on a regular basis.
But afaik there is no built-in solution for this currently. And therefore there is no restore as well.
Something I am planning to do myself at some point. Up until then, because my setup is very static, the described approach does work for me.