Using ZFS Snapshots on rpool/ROOT/pve-1?

[LnxBil]

ok, but what about partition2 / EFI , where proxmox does place the kernels and systemd boot config ?

Due to its definition, there is a non-snapshottable filesystem present there.

if it simply adds another , more recent kernel and does not change anything else, it should not matter, should it ?

Normally, in each kernel line, the top-most version will always be kept, so that should not be an issue for most of the time.

i would also like zfs snapshots as emergency fallback when update goes wrong

Yes, that is a good idea, but please also keep a ZFS-capable rescue system at hand and familiarize with it.


In general:

It is also a good idea to create additional zfs datasets for things like /var/cache, /var/lib/vz and /var/log so that you will not loose those contents on a rollback of your main root-dataset. In addition to that, you should ALWAYS use refreservation with e.g. 1 or 2 GB for those datasets so that a potentially full pool will always have room to write your log- and tempfiles.

 

[diversity]

Yes, that is a good idea, but please also keep a ZFS-capable rescue system at hand and familiarize with it.

appearantly things are still in limbo for root zfs rollback
please see
https://forum.proxmox.com/threads/rescue-mode-with-root-on-zfs-trouble.107172/

It is also a good idea to create additional zfs datasets for things like /var/cache, /var/lib/vz and /var/log

are you sure those directories are not also part of the rpool/ROOT/pve-1 directory? and as such snapshotted along with a root snapshot?

 

[LnxBil]

appearantly things are still in limbo for root zfs rollback

If you do a pool upgrades, yes sure. You always need a rescue system that is capable of reading your pool.

are you sure those directories are not also part of the rpool/ROOT/pve-1 directory? and as such snapshotted along with a root snapshot?

The idea is that they are NOT part of pve-1. They are volatile data and do not belong to the OS.

 

[diversity]

The idea is that they are NOT part of pve-1. They are volatile data and do not belong to the OS

I see, well in that case aren't we going about it coming from a wrong angle?
when I run
mount | grep pve-1
i get
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)

as far as I understood it means that rpool/ROOT/pve-1 is somehow linked to the top most directory ( / ), thus also including /var/cache correct?

Does this mean snapshotting rpool/ROOT/pve-1 is potentially a bad idea? assuming we can even get a rollback to work in the first place.

 

[LnxBil]

top most directory ( / ), thus also including

it includes EVERYTHING that is not a mountpoint itself. All logging, caching etc is in there and that is not a part of the OS. Therefore it is a good idea to have all non-os stuff on additional mountpoints. You can read up about LHS here.

Does this mean snapshotting rpool/ROOT/pve-1 is potentially a bad idea? assuming we can even get a rollback to work in the first place.

You roll back a lot more. That's why I recommend splitting the data into volatile and non-volatile stuff, aswell as separation of concerns.

You container images don't need to be included in OS snapshots, they should stay apart. The same is for logging and caching that is totally irelevant for your OS snapshot will only use up space.

 

[diversity]

I think your suggestion makes a lot of sense.

If only we had a rescue disk that actually was able to rollback the snapshots created using the pve 7.1 iso zfs version.

I'll echo again. Should anyone run into a fix for this then please let the world know.

 

[LnxBil]

I'll echo again. Should anyone run into a fix for this then please let the world know.

Until Proxmox staff fixes this problem (new installer iso with a current ZFS version), you always have the not-so-easy-solution to build your own rescue distribution.

 

[diversity]

thought so. But I am not expecting any love from the devs as they seem to not support this scenario and rather have people use the backup server.
Once back from holiday I'll give this one a try
https://www.complete.org/zfs-rescue-disc/

 

[LnxBil]

But I am not expecting any love from the devs as they seem to not support this scenario and rather have people use the backup server.

But the backup-server will not solve this problem. I still thing it may get better for the next version. It was just bad luck that there was a major incompatible change in the ZFS implementation from the last iso to the newest version in the repository. This may have been (hopefully) a seldom event.

 

[diversity]

uggg. just revisited this zfs rescue disk only to find it last being updated in 2014. so I won't even try.
grrrr. back to square one ;(

 

[LnxBil]

You can always boot any other live linux distribution and try that (ubuntu has ZFS on board, on debian you will have to compile it yourself but it works).

 

[diversity]

yes indeed. Success!! this time!.

I tried once before but got stuck then.
I just tried again with the ubuntu 22.04.1 ISO.

I ran into the same issue as earlier as no pool is available, but this time figured to import it. which worked. rolled back, exported the pool and everything works like a charm.

I am sooo happy at the moment

I still believe that any OS should have this rollback functionality out of the box and newb proof.

 

[LnxBil]

I still believe that any OS should have this rollback functionality out of the box and newb proof.

Unfortunately, not going to happen due to the EFI restriction. FAT is not snapshottable.

 

[aabbccdd]

In fact, it works !! Yeahhh
I'm so happy, I now have a way to easily restaure any snapshoted version of the OS in any kind of situation, in less than 15 minutes (thanks to OVH installer and live network rescue system...)

[...[

I think now the tread can be closed

Hi all.

I've just did something similar.

Why? I wanted to resize 2x cheap 1TB nvme's used in a RAIDZ1 mirror for the rpool pool to take advantage of their space for ZFS caching but one cannot simply resize them so I made a snapshot of the root, reinstalled proxmox and placed the snapshot back in line with what jeanlau did.

My steps:
For those that use the rpool only for the proxmox OS.
Take a snapshot
I'm using zfs-auto-snapshots so there are always plenty fresh ones. Just verify that they are actually there.
Install Promox VE (terminal UI, Debug mode)
Download latest ISO and write to USB stick (used 8.1-1)
Be very mindful of disk selection!! Only select those you need to re-build the rpool pool.
With 500 mb ZRAID1 on 2x 1tb disks
DO NOT REBOOT, press Alt+F3
Install systemd-boot
apt update
apt -y install systemd-boot systemd-boot-efi
apt -y upgrade
Import rpool
zpool import rpool
Delete the volume just installed
zfs destroy rpool/ROOT/pve-1
Import the pool containing the snapshot
zpool import -f rust
#Zpool import -af
Send the snapshot to the root of promox
zfs send -vvv rust/syncoid/rpool/ROOT/pve-1@autosnap_2023-11-28_12:57:39_hourly | zfs receive -vvv rpool/ROOT/pve-1
Export pools
zpool export deadpool
zpool export rpool
Setup chroot
zpool import -f -R /mnt rpool
for i in proc dev sys run ; do mount -o rbind /$i /mnt/mnt/ROOT/pve-1/$i ; done
NB If you get a root shell before the installation, you can simply use /mnt (not /mnt/mnt/ROOT/pve-1)
Check if mounts are done correctly.
mount | grep pve-1
Enter chroot
chroot /mnt/rpool/ROOT/pve-1 /bin/bash
or (if you have not just installed a proxmox system)
chroot /mnt /bin/bash
source /etc/profile
vim /etc/kernel/cmdline # if you need to change anything
systemd-boot: cleanup
proxmox-boot-tool clean
systemd-boot: init EFI partition
# found using root@server:~# lsblk -o +FSTYPE | grep nvme[12] | grep p2
# https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool#3._Finding_potential_ESPs
proxmox-boot-tool init /dev/<EFI partition, usually between 500-1000MB>
proxmox-boot-tool init /dev/<EFI partition, usually between 500-1000MB>
proxmox-boot-tool status
Correct root mountpoint
zfs set mountpoint=/ rpool/ROOT/pve-1
If this command fails and the system hangs on two lines starting with 'EFI' just after the bios has loaded, go back in the installer console, setup chroot again (but with /mnt) import the rpool
zpool import rpool -f
zfs set mountpoint=/ rpool/ROOT/pve-1
zpool export rpool
REBOOT
Import rpool
An emergency shell may be dropped on you
zpool import -f rpool
or import them all
zpool import -fa
REBOOT
DONE.

Sources:
https://pve.proxmox.com/wiki/ZFS:_S...xmox_Boot_Tool#Switching_to_proxmox-boot-tool
https://pve.proxmox.com/wiki/ZFS:_S...iring_a_System_Stuck_in_the_GRUB_Rescue_Shell
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot

I think now the tread can be closed again ;-)

This is known to work. I had problems earlier that appeared difficult to troubleshoot (bios update, nvme firmware update, kernel upgrade, move from grub to systemd-boot, disk swap, cpu scheduler and mode change all in one go). Yes, I got over enthousiastic while having a bit of time on my hands at last. All fixed now though!

Cool article about what is possible (e.g., snapshot before upgrade)
https://oblivious.observer/posts/poc-boot-environments-proxmoxve6/

PS Sorry for the flat list. No time to format it now.

 

[zodiac]

If you don't want to use a rescue disk (or don't have one with a suitable kernel), but if you still have the ability to boot into your system, then the best option would be to roll back from inside the "initramfs". You can probably do this by booting with the "break=bottom" kernel command line option.

But if this is something you expect to do on a regular basis, then messing with the boot loader is tedious. I added a small script to my "initramfs" to automate things:

#!/bin/sh

if [ $PPID -eq 1 ]; then
  # This code snippet runs inside of the initramfs, right after the kernel
  # has mounted the ZFS root file system.
  # If there is a file telling us that a rollback was previously requested,
  # attempt to do so now.
  # N.B.: This code runs in a BusyBox environment. Don't expect to get all of
  # the same powerful commands that a full Linux system has access to. Tread
  # with care when writing shell scripts.
  if [ -r "${rootmnt}/.rollback-zfs" ]; then
    rollback() {
      local label="$1"
      # We only ever attempt to roll back the root filesystem, as it is the only
      # one that is a little tricky to do. Snapshots for other mountpoints
      # should be handled with the tools that Linux and/or PVE provide for this
      # purpose.
      local root="$(awk "/ ${rootmnt////\\/} zfs /"'{ print $1 }' /proc/mounts)"
      # While ZFS has the option to roll back multiple generations of snapshots,
      # we aim to be conservative and will only ever roll back the most recent
      # snapshot. If you need to roll back to an earlier snapshot, you should
      # explicitly destroy the ones that you no longer need.
      local snapshots="$(zfs list -t snapshot "${root}" 2>/dev/null |
                         awk "/^${root////\\/}/"'{ print $1 }')"
      local last="$(echo "${snapshots}" | sed -n 's/[^@]*@//;$p')"
      [ -n "${last}" -a "x${last}" = "x${label}" ] || return
      # Remove the file telling us to perform a rollback operation. This should
      # help avoid boot loops if the rollback is not successful.
      rm -f "${rootmnt}/.rollback-zfs"
      zfs rollback "${root}@${label}" || :
    }
    rollback "$(cat "${rootmnt}/.rollback-zfs")"
    # Again, remove the file telling us to perform rollbacks. This prevents
    # nasty surprises if this file made it into the snapshot.
    rm -f "${rootmnt}/.rollback-zfs"
  fi
elif [ "x$*" = "xprereqs" ]; then
  # Our script can get invoked for several different reasons; one of them
  # would be when the initramfs gets assembled. Don't do anything in that
  # case.
  exit 0
else
  # This is our entry-point into this script, when a user invokes us to
  # manage their snapshots. Perform the required sanity checking and print
  # appropriate messages.
  usage() {
    echo "${0##*/}: [--cancel] <snapshot>" >&2
  }

  # Check for root permissions
  [ "$(id -u)" = 0 ] || { echo "Must run as root user" >&2; exit 1; }
  [ $# = 1 ] || { usage; exit 1; }

  # If invoked with the "--cancel" option, delete the file that controls
  # rollbacks.
  if [ "x$1" = "x--cancel" ]; then
    rm -f /.rollback-zfs
    exit 0
  fi

  # Sanity check that the caller is asking us to remove the most recent
  # snapshot. This information can of course still change several times
  # before the system has actually rebooted, and we will double-check again
  # before performing the actual rollback. But an early check at this
  # place allows us to print helpful messages to the user.
  root="$(awk "/ \/ zfs /"'{ print $1 }' /proc/mounts)"
  [ -n "${root}" ] || {
    echo "Cannot find a root ZFS filesystem" >&2
    exit 1
  }
  r_="$(echo "${root}" | sed 's,/,\/,g')"
  snapshots="$(zfs list -t snapshot "${root}" 2>/dev/null |
               awk "/^${r}/"'{ print $1 }')"
  echo "${snapshots}" | egrep @"$1"'([[:space:]]|$)' >/dev/null 2>&1 || {
    echo "Root filesystem on ${root} does not have a snapshot called $1" >&2
    exit 1
  }
  last="$(echo "${snapshots}" | sed -n 's/[^@]*@//;$p')"
  [ "x$1" = "x${last}" ] || {
    echo "Will only roll back to most recent snapshot. Delete any uneeded" >&2
    echo "snapshots, if you want to roll back to an older one instead." >&2
    exit 1
  }

  # Make sure our script is part of the initramfs for the current kernel.
  if ! [ -r /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs ] ||
      ! cmp "$0" /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs \
            >/dev/null 2>&1; then
    cp "$0" /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs
    update-initramfs -u
    # proxmox-boot-tool refresh
    echo
  fi

  # Explain the next steps that the user needs to take.
  echo "$1" >/.rollback-zfs
  cat <<EOF
The system will roll back to "$1" upon the next reboot.

You must initiate this reboot yourself (e.g. by running "/usr/sbin/reboot").

After the system has booted again, the snapshot will NOT be removed
automatically. If you no longer need it, you must execute
"zfs destroy ${root}@$1" yourself.
EOF
fi

Save this code to something like "/usr/local/sbin/rollback-zfs". You can then invoke it with the name of a ZFS snapshot and reboot the system. The "initramfs" will perform the requested rollback operation for you.

 

[aabbccdd]

If you don't want to use a rescue disk (or don't have one with a suitable kernel), but if you still have the ability to boot into your system, then the best option would be to roll back from inside the "initramfs". You can probably do this by booting with the "break=bottom" kernel command line option.

But if this is something you expect to do on a regular basis, then messing with the boot loader is tedious. I added a small script to my "initramfs" to automate things:

#!/bin/sh

if [ $PPID -eq 1 ]; then
  # This code snippet runs inside of the initramfs, right after the kernel
  # has mounted the ZFS root file system.
  # If there is a file telling us that a rollback was previously requested,
  # attempt to do so now.
  # N.B.: This code runs in a BusyBox environment. Don't expect to get all of
  # the same powerful commands that a full Linux system has access to. Tread
  # with care when writing shell scripts.
  if [ -r "${rootmnt}/.rollback-zfs" ]; then
    rollback() {
      local label="$1"
      # We only ever attempt to roll back the root filesystem, as it is the only
      # one that is a little tricky to do. Snapshots for other mountpoints
      # should be handled with the tools that Linux and/or PVE provide for this
      # purpose.
      local root="$(awk "/ ${rootmnt////\\/} zfs /"'{ print $1 }' /proc/mounts)"
      # While ZFS has the option to roll back multiple generations of snapshots,
      # we aim to be conservative and will only ever roll back the most recent
      # snapshot. If you need to roll back to an earlier snapshot, you should
      # explicitly destroy the ones that you no longer need.
      local snapshots="$(zfs list -t snapshot "${root}" 2>/dev/null |
                         awk "/^${root////\\/}/"'{ print $1 }')"
      local last="$(echo "${snapshots}" | sed -n 's/[^@]*@//;$p')"
      [ -n "${last}" -a "x${last}" = "x${label}" ] || return
      # Remove the file telling us to perform a rollback operation. This should
      # help avoid boot loops if the rollback is not successful.
      rm -f "${rootmnt}/.rollback-zfs"
      zfs rollback "${root}@${label}" || :
    }
    rollback "$(cat "${rootmnt}/.rollback-zfs")"
    # Again, remove the file telling us to perform rollbacks. This prevents
    # nasty surprises if this file made it into the snapshot.
    rm -f "${rootmnt}/.rollback-zfs"
  fi
elif [ "x$*" = "xprereqs" ]; then
  # Our script can get invoked for several different reasons; one of them
  # would be when the initramfs gets assembled. Don't do anything in that
  # case.
  exit 0
else
  # This is our entry-point into this script, when a user invokes us to
  # manage their snapshots. Perform the required sanity checking and print
  # appropriate messages.
  usage() {
    echo "${0##*/}: [--cancel] <snapshot>" >&2
  }

  # Check for root permissions
  [ "$(id -u)" = 0 ] || { echo "Must run as root user" >&2; exit 1; }
  [ $# = 1 ] || { usage; exit 1; }

  # If invoked with the "--cancel" option, delete the file that controls
  # rollbacks.
  if [ "x$1" = "x--cancel" ]; then
    rm -f /.rollback-zfs
    exit 0
  fi

  # Sanity check that the caller is asking us to remove the most recent
  # snapshot. This information can of course still change several times
  # before the system has actually rebooted, and we will double-check again
  # before performing the actual rollback. But an early check at this
  # place allows us to print helpful messages to the user.
  root="$(awk "/ \/ zfs /"'{ print $1 }' /proc/mounts)"
  [ -n "${root}" ] || {
    echo "Cannot find a root ZFS filesystem" >&2
    exit 1
  }
  r_="$(echo "${root}" | sed 's,/,\/,g')"
  snapshots="$(zfs list -t snapshot "${root}" 2>/dev/null |
               awk "/^${r}/"'{ print $1 }')"
  echo "${snapshots}" | egrep @"$1"'([[:space:]]|$)' >/dev/null 2>&1 || {
    echo "Root filesystem on ${root} does not have a snapshot called $1" >&2
    exit 1
  }
  last="$(echo "${snapshots}" | sed -n 's/[^@]*@//;$p')"
  [ "x$1" = "x${last}" ] || {
    echo "Will only roll back to most recent snapshot. Delete any uneeded" >&2
    echo "snapshots, if you want to roll back to an older one instead." >&2
    exit 1
  }

  # Make sure our script is part of the initramfs for the current kernel.
  if ! [ -r /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs ] ||
      ! cmp "$0" /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs \
            >/dev/null 2>&1; then
    cp "$0" /usr/share/initramfs-tools/scripts/local-bottom/rollback-zfs
    update-initramfs -u
    # proxmox-boot-tool refresh
    echo
  fi

  # Explain the next steps that the user needs to take.
  echo "$1" >/.rollback-zfs
  cat <<EOF
The system will roll back to "$1" upon the next reboot.

You must initiate this reboot yourself (e.g. by running "/usr/sbin/reboot").

After the system has booted again, the snapshot will NOT be removed
automatically. If you no longer need it, you must execute
"zfs destroy ${root}@$1" yourself.
EOF
fi

Save this code to something like "/usr/local/sbin/rollback-zfs". You can then invoke it with the name of a ZFS snapshot and reboot the system. The "initramfs" will perform the requested rollback operation for you.

Fancy!

 

[ivenae]

Cool, i will try that in the future.
Maybe i will build an environment that snapshot before an update and checks for network connection afterwards and automatically rollback and reboot if the system has no network connection or boots into initramfs after the upgrade.