Using ZFS Snapshots on rpool/ROOT/pve-1?

[mattlach]

Hey all,

Personally I always like to take precautions whenever I upgrade something to make sure that package updates don't break anything for me.

I've been wondering, on a ZFS based install, is snapshotting rpool/ROOT/pve-1 a good way of accomplishing this?

In theory, I could run the snapshot, do the "apt-get dist-upgrade" and if something goes wrong, just restore the snapshot.

There are potential problems here though, as I see them:

1.) When I do this, I am performing a snapshot on a running system, which probably isn't the best choice. During restore, I'd be restoring a booted state, and then have to reboot the system, which might cause some sort of problems as the running upgraded system tries to shut down and write to the disk as part of it's shutdown process.

2.) Are the aspects of apt updates that are made to areas that are not in rpool/ROOT/pve-1?

I'd appreciate any thoughts on this.

 

[wolfgang]

Hi,

The snapshot should be not the problem.
ZFS itself is consistent and to flush the cache you can use sync before.

If you must rollback in case of an error, this can only done by Grub.

2. No

 

[jeanlau]

Hi there,

I am like mattlach and I try, like him, to use snapshots in order to get a previous version when I upgrade the systems, also I would like so much to be able to make cross backups between my pve hosts in case of a complete server crash...
I'm trying to obtain this result, and it works !! No problem to make snapshots or even to transfer them across machines, maybe I will even script that later or try to find one that already exists and suit my needs since they are already a lot of snapshots management scripts...
I am surprised there is no more people interested by this kind of setup...

zfs snap rpool/ROOT/pve-1@snap1
zfs snap rpool/ROOT/pve-1@snap2
zfs snap rpool/ROOT/pve-1@snap3
zfs snap rpool/ROOT/pve-1@snap4

from host 1 :
zfs send rpool/ROOT/pve-1@snap1 | ssh host2 zfs receive rpool/backups/cluster/rootfs-host1
and then,
zfs send rpool/ROOT/pve-1@snap1 | ssh host2 zfs receive rpool/backups/cluster/rootfs-host1
and so on... or even,
zfs send -R rpool/ROOT/pve-1@snap4 | ssh host2 zfs receive -F rpool/backups/cluster/rootfs-host1

from host 2 :
zfs send rpool/ROOT/pve-1@snap1 | ssh host3 zfs receive rpool/backups/cluster/rootfs-host2
and then,
zfs send rpool/ROOT/pve-1@snap1 | ssh host3 zfs receive rpool/backups/cluster/rootfs-host2
and so on... or even,
zfs send -R rpool/ROOT/pve-1@snap4 | ssh host3 zfs receive -F rpool/backups/cluster/rootfs-host2

from host 3 :
zfs send rpool/ROOT/pve-1@snap1 | ssh host1 zfs receive rpool/backups/cluster/rootfs-host3
and then,
zfs send rpool/ROOT/pve-1@snap1 | ssh host1 zfs receive rpool/backups/cluster/rootfs-host3
and so on... or even,
zfs send -R rpool/ROOT/pve-1@snap4 | ssh host1 zfs receive -F rpool/backups/cluster/rootfs-host3

But, (of course, there is a but..) I am trying since days... without any luck when I try to rollback or when I simulate the crash of one server and I transfer back a snapshot or fs, I'am completely unable to use it.

I think I'm close but if someone could help me to fill the gap between where I am in the process and the success, I would be extremely grateful.

Also, I saw you mentioning about grub, could you please explain how to achieve this goal ? I'm sorry if it's obvious, I thought it might be but in fact it seem to be tricky for me ! I searched hours and days on google, mailing lists, forum and didn't found any good tutorial or procedure that provide an easy way to achieve this...

Recently, you added a super wonderful great option to manage VM zfs replication across the hosts but in a disaster recovery situation, it would be so great to have an option to also backup (and restore of course !) the pve hosts themselves too...

Please help, thank you very much

 

[jeanlau]

Hi,

I just wanted to add that I successfully tested rollback !!
You just have to take snapshot before any important admin task :

zfs snap rpool/ROOT/pve-1@installOK
zfs snap rpool/ROOT/pve-1@MAJ1
zfs snap rpool/ROOT/pve-1@MAJ2

Then, if need to go back, boot on a live CD with ZFS on linux support (ex : proxmox install cd, rescue mode) or install it, import the pool then you just enter the rollback command :

zfs rollback rpool/ROOT/pve-1@MAJ1
=>> Blabla not happy, there is newer snapshot... use -f to remove older snapshots...
zfs rollback -f rpool/ROOT/pve-1@MAJ1
removing rpool/ROOT/pve-1@MAJ2, blabla, OK
reboot

Now reboot and voilà !

Now I still can't don't find documentation on the net about how to get back a root fs backup on an another machine and be able to reboot on this backup... I will try more...
Please if someone knows how to, tell me !!

Thanks

 

[jeanlau]

In fact, it works !! Yeahhh
I'm so happy, I now have a way to easily restaure any snapshoted version of the OS in any kind of situation, in less than 15 minutes (thanks to OVH installer and live network rescue system...)

In case someone interested, I paste my little procedure I made for myself (sorry for comments in French, use google translate ):

Procédure de restore PVE depuis ZFS:

- Réinstall depuis l'installateur OVH ( template Install-PVE5-ZFS-Base)

- Reboot sur le rescue BSD pour récuperer le fs (Debian rescue ça plante à la récup...)

- importer le pool proprement :
zpool import -f -N -R /mnt rpool

- supprimer le dataset de base de l'install :
zfs destroy -r rpool/ROOT/pve-1 (-r si il y a des snap)

- Sur le serveur source, revenir a un état voulu et virer les snap sauf le premier !! (mais pas sûr que ce soit nécessaire... ) :
root@srv1:~# zfs rollback -r rpool/bkps/rootfs-srv2@snap4
root@srv1:~# zfs destroy rpool/bkps/rootfs-srv2@snap2
root@srv1:~# zfs destroy rpool/bkps/rootfs-srv2@snap3
root@srv1:~# zfs destroy rpool/bkps/rootfs-srv2@snap4

- renvoyer le backup du root fs depuis un serveur de backup :
ssh srv2 (pour éviter erreur de clé)
(erreur de clé ssh ? sed -i '5d' ~/.ssh/known_hosts ou encore ssh-keygen -f "/root/.ssh/known_hosts" -R zeus.ipgenius.fr)
zfs send -vvv rpool/bkps/rootfs-zeus@snap1 | ssh zeus.ipgenius.fr zfs receive -vvv rpool/ROOT/pve-1

- exporter proprement le pool :
zpool export rpool

- changer les mountpoint (peut etre fait sous grub en cas d'oubli...) :
zfs set mountpoint=/ rpool/ROOT/pve-1

Less than 15min to come back from any situation

I think now the tread can be closed

 

[liquidox]

Just found this thread, so amazing! Going to use this on my new build. Not being afraid of upgrades breaking something would be superb.

 

[diversity]

I liked your solution even though not fully understanding it yet I figured that it seems to work otherwise there would have been questions already.

I am going to dive in and see how far I get.

I think a procedure like this should be a standard feature as it is sooooo convenient.

 

[cserem]

If you must rollback in case of an error, this can only done by Grub.

Hello!

I can't find anything regarding this in the Proxmox VE administration Guide, could you elabore the zfs rollback process with grub?

Thanks

 

[gomu]

Hello,
I was able to use this technique successfully on OVH dedicated servers too. I'm eager to us it on in-house proxmox systems.
ZFS rules really !

 

[diversity]

Could you please try explaining once in a different manner than @jeanlau did the steps you took? I am not getting through his explenation

 

[diversity]

Hello!

I can't find anything regarding this in the Proxmox VE administration Guide, could you elabore the zfs rollback process with grub?

Thanks

@wolfgang , I would very much would like to second @cserem 's request

 

[gomu]

Hello,

I don't see any relation between grub and ZFS -on-root, as far as rollback is concerned.

As soon as you choose zfs as install option, you get later the ability to rollback any system change you do. Software upgrade, configuration change, file removal, etc.

1. Before any modification, do : zfs snapshot rpool/ROOT/pve-1@snapshot_name
2. If you want to rollback, do : zfs rollback rpool/ROOT/pve-1@snapshot_name; reboot (better boot in a rescue environment before, but it can work straight from your normal boot too).

Useful too : apt install zfs-auto-snapshot; it will create "recent", hourly, daily, weekly and monthly snapshots with auto purging.

Grub itself has two parts :

• one on the bootblock : this one is not covered by ZFS, if you install Windows on a different disk than you proxmox installation for example, you will loose the ability to boot into proxmox and you will need to restore linux boot by your own means
• the other in the filesystem, like configuration files, etc : this one is covered by zfs (as any other change you make on your zfs-enabled proxmox system) and if mistakes are done, it can be easily rollbacked as explained earlier.

Hope it's clearer ?

 

[diversity]

I am checking it out sooner or later

 

[gomu]

As far as zfs-auto-snapshot package, I found better to disable it entirely zfs set com.sun:auto-snapshot=false rpool first. This property will get inherited to all FS.
Then I activated it only for the proxmox system zfs set com.sun:auto-snapshot=true rpool/ROOT/pve-1.
Without this, it would make regular snapshots of your swap (not really useful...) and all containers (don't know about VMs but probably also, which may be confusing if you have backups or replication in place).
If you want it on containers, you can enable them selectively with zfs set com.sun:auto-snapshot=true rpool/data

 

[diversity]

thank you for your more detailed analysis.

Why can't one simply snapshot the root/rpool?

 

[gomu]

Of course you can.
I feel the result is a bit messy, that's why I prefer to do on a as-needed basis

 

[diversity]

But in theory having the uppermost part of a zfs pool referted should not be messy?

What in practicse shouls i watch out for?

 

[gomu]

By default, zfs-auto-snapshot + inherit capacity of ZFS will cascade snapshots to every entry point.
By messy, I meant it will then keep snapshots for root/rpool, swap, possibly every container, etc. Thats' a lot of snapshots over time !
In, my case I wanted snapshots only of my / filesytem, hence rpool/ROOT/pve-1 property tuning, not all the others.

 

[diversity]

Thank you @gomu ,

@developers is there anything down the roadmap to level the functionality of @jeanlau his workflow between nodes?

 

[RolandK]

ok, but what about partition2 / EFI , where proxmox does place the kernels and systemd boot config ?

if it simply adds another , more recent kernel and does not change anything else, it should not matter, should it ?

i would also like zfs snapshots as emergency fallback when update goes wrong