[TUTORIAL] PVE (rootfs) with autosnapshot, boot into snapshot and easy rollback on BTRFS Raid1 for Bios or UEFI

F

floh8

Renowned Member
Jul 27, 2021
1,255
166
88
Occasion:
Like my one mentioned in the zfs boot thread my intention was to use BTRFS as root filesystem because the resource finger print is smaller. While my testing time for this thread some negative experiences with BTRFS comes up, but nevertheless my one would prefer btrfs over zfs for rootfs. This solution combines the tools snapper, grub-btrfs and overlayroot (optional). There are many btrfs snapshot tools for the "easy way" out there, but for production server the best is snapper from my point of view. What my one dont need to mention is that a BTRFS Raid 1 solution with my demands your one cant find in the internet already.

Table of Contents:
  1. My goals for this implementation
  2. Troubles while creation
  3. Requirements
  4. Configuration steps necessary 4 both Bios or UEFI boot
  5. UEFI install
  6. BIOS install
  7. Rollback possibilities
  8. Back to proxmox-boot environment from UEFI
  9. Back to proxmox-boot environment from Bios
  10. Sources

1. My goals for this implementation:
  1. as little change as possible to the original installation
  2. changes to new bootloader are based of a PVE ISO standard installation
  3. in a few steps back to the original PVE boot environment
  4. kernel image files stored on the root partition for snapshoting
  5. kernel pinning possible
  6. easy snapshot rollback, also when OS does not boot anymore
  7. snapshot boot menu
  8. MemTest im Bootmenu available
  9. system boots even if one drive is offline
  10. GRUB is used for all boot combinations except no loader is needed
The new btrfs boot environment and tools are then as follows:

BTRFS.png

2. Troubles while creation
  1. snapper creates only readonly snapshots
    • this produces 2 problems when booting in a snapshot: the first is that pve services could not start error free, so a temporary test run is not possible; the second is that if one wants to execute "update-grub" directly after roolback this is not possible
    • SOLUTION: using a subvolume "@grub" for /boot/grub and the debian packet "overlayroot"
  2. snapper create a subvolume /.snapshots as targets for its snapshots per default
    • btrfs excludes subvolumes in its snapshots, so when one boots into a snapshot for rolling back or boots a rolled back snapshot, snapper is not able to access its snapshots anymore
    • SOLUTION: use mountpoint /.snapshots for snapshots subvolume
  3. GRUBs environment variable "prefix" for grub.cfg is hardcoded at grub-install time
    • that means: when one boot into a rolled back snapshot and change the grub.cfg the changes are not visible because GRUB looks still to the filesystem path from where it was installed
    • SOLUTION: using a subvolume "@grub" for /boot/grub
  4. PVE 8 installation with UEFI configure a mountpoint for sda2 in fstab
    • this goals in a boot error when try to boot a degraded raid
    • SOLUTION: comment out this fstab entry
  5. PVE default installation does'nt set mount option "degraded" for "/" nor kernel parameter
    • your system will run into a early boot error when its degraded (one disk offline)
    • My one posted a bug to Proxmox, therefor they can add this parameter for a next big release. The reaction of Proxmox was the nonsensical my one ever read in a IT thread. Proxmox answered that to use this parameter is dangerous because btrfs does'nt offer an integrated notification service like zfs does. So my one suggest to eliminate the btrfs raid1 option in the installation wizard because it makes no sense when one can not reboot with a raid error. The bugzilla post was deactivated or deleted. One cant find it anymore.
    • SOLUTION: add parameter "degraded" to fstab and kernel call
    • My research revealed a raid1 error with actual kernels that booting several times into a degraded raid 1 can set the btrfs filesystem to readonly. My tests with PVE 9 showed no problem with that but my one noticed that the command "update-grub" does not function anymore. So this is, of course, no production ready state although debian have been tagged btrfs raid 1 as stable for years. My opinion is that everyone have to decide for his one whether to use btrfs as root filesystem or not. To combine mdadm raid with btrfs is less useful than using zfs.

3. Requirements:
A normal PVE 8 or 9 BTRFS (Raid1) installation. The PVE install disks are sda and sdb.


4. Configuration steps necessary 4 both Bios or UEFI boot:

  • create nesseccary subvolumes
btrfs subvolume snapshot / /@rootfs​
btrfs subvolume create /@snapshots​
btrfs subvolume create /@grub​
  • make rootfs to the default subvolume to boot from
btrfs su sh /@rootfs​
btrfs subvolume set-default [ID] /​
  • change fstab to the new layout (ATTENTION: line for /.snapshots and ESP mount is comment out)
nano /@rootfs/etc/fstab​

UUID=722b146c-d11f-48dd-91c4-a4448792041f / btrfs subvol=@rootfs,defaults,degraded,compress 0 1​
#UUID=722b146c-d11f-48dd-91c4-a4448792041f /.snapshots btrfs subvol=@snapshots,defaults,compress 0 1​
UUID=722b146c-d11f-48dd-91c4-a4448792041f /boot/grub btrfs subvol=@grub,defaults,compress 0 1​

#UUID=CEA1-7F15 /boot/efi vfat defaults 0 1​
  • send content from grub to new subvolume
mv /@rootfs/boot/grub/* /@grub/​
  • copy local-btrfs to new location
rmdir /@rootfs/var/lib/pve/local-btrfs​
btrfs su sn /var/lib/pve/local-btrfs /@rootfs/var/lib/pve/local-btrfs​
btrfs su de /var/lib/pve/local-btrfs​
  • set "degraded" to kernel command line
nano /@rootfs/etc/default/grub​
​
GRUB_CMDLINE_LINUX_DEFAULT="rootflags=degraded"​
  • reboot your system
  • delete old rootfs data of subvolid=5
mkdir /mnt/btrfsroot​
mount -o subvolid=5 /dev/sda3 /mnt/btrfsroot​
cd /mnt/btrfsroot​
shopt -s extglob​
rm -rf !(@*)​
shopt -u extglob​
cd /​
  • install addional packets
apt update && apt install build-essential git inotify-tools snapper overlayroot​
  • configure snapper to your needs
snapper -c root create-config /​
btrfs su delete /.snapshots​
mkdir /.snapshots​
nano /etc/snapper/configs/root​
nano /etc/default/snapper​
  • comment in line with "/.snapshots" in fstab
  • install grub-btrfs
git clone https://github.com/Antynea/grub-btrfs.git​
cd grub-btrfs​
make install​
systemctl enable grub-btrfsd​
  • configure grub-btrfs to your needs
nano /etc/default/grub-btrfs/config​

GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@rootfs")​
  • [optional] configure overlayroot
nano /etc/default/grub-btrfs/config​
GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="overlayroot=tmpfs:recurse=0"​


5. UEFI install
​
1. Configuration steps necessary 4 both PVE 8 + 9​
​
mkdir /boot/efi/sda2​
mkdir /boot/efi/sdb2​
mount /dev/sda2 /boot/efi/sda2​
mount /dev/sda2 /boot/efi/sda2​

2. only for PVE 9​
  • delete ESP-UUIDs in proxmox-boot-uuids file
true > /etc/kernel/proxmox-boot-uuids​
  • install grub
grub-install.real --efi-directory=/boot/efi/sda2 --bootloader-id=grub-sda /dev/sda2​
grub-install.real --efi-directory=/boot/efi/sdb2 --bootloader-id=grub-sdb /dev/sdb2​
update-grub​
  • make new grub entries to the default Bootloader, when all UEFI profiles are cleared
cp /boot/efi/sda2/EFI/grub-sda/* /boot/efi/sda2/EFI/BOOT/​
cp /boot/efi/sda2/EFI/grub-sda/grubx64.efi /boot/efi/sda2/EFI/BOOT/BOOTx64.EFI​
cp /boot/efi/sdb2/EFI/grub-sdb/* /boot/efi/sdb2/EFI/BOOT/​
cp /boot/efi/sdb2/EFI/grub-sdb/grubx64.efi /boot/efi/sdb2/EFI/BOOT/BOOTx64.EFI​

3. only for PVE 8​
  • change grub variable "prefix" in grub.cfg on both ESPs
nano /boot/efi/sda2/EFI/proxmox/grub.cfg​
nano /boot/efi/sdb2/EFI/proxmox/grub.cfg​
​
set prefix=($root)'/@grub'​
  • update grub-menu
update-grub​
​
6. BIOS install
  • [only PVE 9] delete ESP-UUIDs in proxmox-boot-uuids file
true > /etc/kernel/proxmox-boot-uuids​
  • install grub
grub-install.real /dev/sda​
grub-install.real /dev/sdb​
  • update grub-menu
update-grub​

7. Rollback possibilities
With this solution it does`nt matter if the system is already booted into the wanted sapshot or not. One can rollback in both situations. The only difference is that if one is still logged in the old subvolume one have to give the command "update-grub" after reboot into the target snapshot.

1. snapper -a classic rollback [id des snapshots]​
2. update-grub​


8. Back to proxmox-boot environment from UEFI
For a PVE 8 installation this solution changed only the path for the grub environment variable "prefix". So nothing must changed back.

For PVE 9​
  • readd ESP UUIDs in /etc/kernel/proxmox-boot-uuids; one find them with "blkid"
nano /etc/kernel/proxmox-boot-uuids​
  • deactivate the both grub UEFI entries; one find the IDs in the 1. column from the output of command "efibootmgr"
efibootmgr -Ab [ID grub-sda]​
efibootmgr -Ab [ID grub-sdb]​
  • copy systemd efi file back to standard BOOT
mount /dev/sda2 /boot/efi/sda2​
mount /dev/sda2 /boot/efi/sda2​
cp /boot/efi/sda2/EFI/systemd/systemd-bootx64.efi /boot/efi/sda2/EFI/BOOT/BOOTX64.EFI​
cp /boot/efi/sdb2/EFI/systemd/systemd-bootx64.efi /boot/efi/sdb2/EFI/BOOT/BOOTX64.EFI​

9. Back to proxmox-boot environment from Bios
For a PVE 8 installation this solution changed only the path for the grub environment variable "prefix". So nothing must changed back.

For PVE 9​
  • readd ESP UUIDs in /etc/kernel/proxmox-boot-uuids; one find them with "blkid"
nano /etc/kernel/proxmox-boot-uuids​
  • initialize the grub environement new
proxmox-boot-tool reinit​


10. Sources
 
Last edited:
  • Like
Reactions: Domino
Nice! It's been years since I last used BTRFS with anger, will give your script a run in my lab, thanks! We need to see more instructional step-by-step writeups on here, more resources the better.
 
You must log in or register to reply here.
Top Bottom