Installing on top of Debian with Systemd-boot and BTRFS

[t0mz]

I am migrating my server from Arch to Debian/PVE. Given that there are quite a few Docker services to migrate, I will dualboot until I successfully moved all services.
What makes matters a little non standard is that I use systemd-boot and btrfs, which is why I thought I'd double check here before starting adding PVE to Debian.

Having Debian and Arch coexist turned out to be easier than expected, I created a new subvol @debian, did a debootstrap, installed the kernel files and was pretty much done with my dual boot setup.

Now, after reading this https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_11_Bullseye and https://pve.proxmox.com/wiki/Host_Bootloader I am a bit unsure about the sequence of events when it comes to installing PVE:
The Debian installation is on Kernel 5.10

1. First I add the PVE repos to /etc/apt/sources.list.d/pve-install-repo.list
2. Then I do an `apt install pve-kernel-5.15`
At this point my systemd-boot config will be still pointing to 5.10, the suggested reboot will break the system

3. Should I now run `proxmox-boot-tool refresh` to make sure an approbriate systemd-boot config file is created?
4. Continue on with the PVE installation -> Reboot, apt install proxmox-ve…

And assuming all of this works out as expected, do I need to run `proxmox-boot-tool refresh` with every kernel update or will the PVE install add the necessary hooks to automate this?
In Proxmox VE proxmox-boot-tool is used to keep the configuration on the ESPs synchronized., doesn't say who is using the tool, me manually, or PVE automatically.

 

[fabian]

proxmox-boot-tool will be called automatically on kernel changes, just like other distros hook their initrd tools into that part of package management.

 

[t0mz]

proxmox-boot-tool will be called automatically on kernel changes, just like other distros hook their initrd tools into that part of package management.

Excellent! Thanks for confirming Fabian. Sounds like I am good to upgrade like proposed above?

 

[fabian]

well, it's a bit more complicated if you want to use a single disk and single ESP to dual-boot, as that is not really tested. I assume you are capable of manually fixing stuff if you encounter issues given your setup and the fact that you are (also) using Arch

the following should configure your ESP to use proxmox-boot-tool with systemd-boot (provided you are booted using EFI):
- boot into Debian/PVE
- unmount the ESP
- remove any automounting of the ESP (p-b-t will mount the ESP(s) when needed)
- run proxmox-boot-tool init /dev/XXX where XXX is your ESP block dev

it should output something similar to this:

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="4FD8-48EA" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme0n1" MOUNTPOINT=""
Mounting '/dev/nvme0n1p2' on '/var/tmp/espmounts/4FD8-48EA'.
Installing systemd-boot..
Created "/var/tmp/espmounts/4FD8-48EA/EFI/systemd".
Created "/var/tmp/espmounts/4FD8-48EA/loader".
Created "/var/tmp/espmounts/4FD8-48EA/loader/entries".
Created "/var/tmp/espmounts/4FD8-48EA/EFI/Linux".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/4FD8-48EA/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/4FD8-48EA/EFI/BOOT/BOOTX64.EFI".
Random seed file /var/tmp/espmounts/4FD8-48EA/loader/random-seed successfully written (512 bytes).
Successfully initialized system token in EFI variable with 512 bytes.
Created EFI boot entry "Linux Boot Manager".
Configuring systemd-boot..
Unmounting '/dev/nvme0n1p2'.
Adding '/dev/nvme0n1p2' to list of synced ESPs..
Refreshing kernels and initrds..
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/4FD8-48EA
        Copying kernel and creating boot-entry for 5.15.104-1-pve
        Copying kernel and creating boot-entry for 6.1.14-1-pve
        Copying kernel and creating boot-entry for 6.1.15-1-pve

the first part puts PVE's copy of systemd-boot on the ESP (I am not sure whether that re-uses the one from Arch if it encounters one, or whether it will error out.. it does work fine multiple times in a row though, which could be a good sign )
the second part then copies the kernels and initrds and adds loader entries for them - this will be triggered again if you update/install/remove kernels, and can be manually triggered with proxmox-boot-tool refresh as well.

 

[t0mz]

I see, this is very helpful!
From your output I can see that most of what the tool is doing is already there, except these two:
Created EFI boot entry "Linux Boot Manager". Adding '/dev/nvme0n1p2' to list of synced ESPs..
I keep copies of the ESP so I can happily play around with things, as long as I don't reboot.
If an init overwrites the ESP, I'll stick to manual while dual booting, manually updating my existing Debian.conf to point to the pve kernel.
I also think I'll go with 6.1.14 right away because there were quite some btrfs enhancements in the kernel recently, and it's closer to what Arch is using as well.

I think I am all set! Thanks for your help again.
I'll come back with an update once done to close this off for the benefit of other readers.

 

[fabian]

sure, manual should be fine as well proxmox-boot-tool does nothing if you don't init an ESP using it, and initrds are still generated like always by initramfs-tools, so you can point your bootloader at the PVE kernel and initrd however you want.

 

[t0mz]

So far so good. Migration worked, dual boot is working and PVE is up and running.
There were some error messages during kernel install and setting up of proxmox-boot-tool related to not being able to create a link, which makes sense given that my ESP is vfat.

Next step is to figure out how to best integrate the local btrfs raid into the docker vm I am about to create. According to ChatGPT I should use the virtiofsd driver, but that's a different subject which I need to read up upon before I bother anyone here

 

[fabian]

So far so good. Migration worked, dual boot is working and PVE is up and running.
There were some error messages during kernel install and setting up of proxmox-boot-tool related to not being able to create a link, which makes sense given that my ESP is vfat.

PVE ESPs are also vfat formatted (it's the only filesystem that has firmware support everywhere) if you still have it (e.g., /var/log/apt/term.log) the messages might be interesting, maybe we can improve some error handling somewhere..

Next step is to figure out how to best integrate the local btrfs raid into the docker vm I am about to create. According to ChatGPT I should use the virtiofsd driver, but that's a different subject which I need to read up upon before I bother anyone here

virtiofsd is on our TODO list - @MarkusF recently sent some patches and we are working on packaging the new (Rust-based) version of virtiofsd: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056487.html

but the easiest way to use your btrfs might be to create a subvol for your guest volumes and set that up as BTRFS-type storage in PVE, and then just allocating volumes by adding virtual hard disks - the VM just sees a block device (and can use whatever filesystem it wants inside). the BTRFS plugin is still experimental, but basic features are there (and the missing parts are mainly related to clustering and migration, not relevant for a single node).

 

[t0mz]

it's the only filesystem that has firmware support everywhere

Yup, I know, but wasn't sure if PVE maybe does something unusual here
The log file is still around and I am happy to share. You can find those errors at the very end of the log.
ln: failed to create hard link '/boot/initrd.img-6.1.15-1-pve.dpkg-bak' => '/boot/initrd.img-6.1.15-1-pve': Operation not permitted

 

[fabian]

and what filesystem is /boot? because (in Debian) the ESP is usually only mounted in /boot/efi, with /boot being either a separate partition or just a dir on / of course, if you mount the ESP on /boot, you don't have to keep two copies of the kernels/initrds, but I am not sure how much dpkg and other tools like VFAT..

 

[t0mz]

Ah, that's it. /boot is VFAT all the way down to /boot/EFI.
I can't really remember why though. This server used to be on GRUB, later converted to Sytemd-Boot and, aside from daily kernel updates, I haven't touched this part of the system for the last 4 years or so.

 

[t0mz]

by adding virtual hard disks

I came to the conclusion that I am not yet ready to migrate over to PVE. I think need to wait for virtiofsd support.

My showstopper is 4TB of Nextcloud data. Without virtiofsd support the only way I think I can make this work, is by creating a 4TB VM or add a 4TB raw disk to the VM.
And then there is another 3TB directory on this raid array which is shared out via Samba…which I could probably tackle with a dedicated Samba container or create a 11TB disk for both to match the size of the raid array.

With virtiofsd support I would simply copy the (small) Nextcloud Docker-Compose folder, to the VM map the raid array in the same location as before and be back in business (at least that's my theory )

Keeping things separate whilst still sharing space is one of the advantages of btrfs subvolumes vs. partitions which made me switch to btrfs back then. Going fixed disk images on top of the btrfs layer for my Docker VM would sort of reintroduce these space limitations again. Resizing images of that size will take ages.