Start At Boot- is there a script?

[binarian]

Hi all,

I have a question about the Start At Boot functionality. I have a fully-functional proxmox 6.1-11 machine that, until this morning, was running everything off of an unencrypted zfs dataset. As of this morning, I have moved everything over to an encrypted zfs dataset, and things continue to run smoothly. However, as I have required a passphrase to be entered to unlock the dataset, containers and VMs of course can no longer autostart when the node boots.

So, my question: Is there a script that runs that determines what order things need to start in, and whether to start them at all? What is that script? I have my own script I just wrote to handle asking for the encryption passphrase and mounting of the dataset, and I'd like to kick off this autostart script so that I don't need to manually start things after mounting the filesystem.

Thanks

 

[aaron]

Without looking at the code, I assume that it is not a simple script that you can call from the CLI but baked in more deeply. You could run qm start VMID and pct start VMID from your script to automatically start the VMs on the encrypted dataset once you unlocked it.

 

[binarian]

Without looking at the code, I assume that it is not a simple script that you can call from the CLI but baked in more deeply. You could run qm start VMID and pct start VMID from your script to automatically start the VMs on the encrypted dataset once you unlocked it.

Baaahhhh I was afraid of that. That requires knowing in advance what the ordering and IDs are supposed to be. Still, good to know there's an easy way to start them in general, thanks.

Anyone else have any suggestions?

 

[H4R0]

Baaahhhh I was afraid of that. That requires knowing in advance what the ordering and IDs are supposed to be. Still, good to know there's an easy way to start them in general, thanks.

Anyone else have any suggestions?

just replicate the start order yourself, sleep in between and you have the same setup. Shutdown will not be handled tho.

Personally i use luks instead of zfs encrypt, it offers way more features, had many security audits, uses way stronger encryption and can be hardened against brute force attempts. It can easily handle 2500mb/sec and does not interfere with proxmox setup, everything works like before. Also offers ssh remote decrypt. And most importantly it encrypts everything including rpool. If you have a zfs raid1 install you can even live encrypt without taking the node down.

 

[binarian]

just replicate the start order yourself, sleep in between and you have the same setup. Shutdown will not be handled tho.

Personally i use luks instead of zfs encrypt, it offers way more features, had many security audits, uses way stronger encryption and can be hardened against brute force attempts. It can easily handle 2500mb/sec and does not interfere with proxmox setup, everything works like before. Also offers ssh remote decrypt. And most importantly it encrypts everything including rpool.

It looks like that's what I'll have to do.

As for encryption, this machine is already an abomination (both a raid controller and an HBA, because apparently an r710 won't boot from an unsupported HBA). I'm not in any high-security-risk scenario; I just want encryption so that in the event of a drive failure destroying the disk is less of an hassle. Since I don't have encryption set up on the root drives I'll just smash the disks when they die.

 

[kraut.hosting]

We have also some interest into this so better to continue then start a new thread
For the scope of using ZFS encrypted dataset for all VMs a proper solution is desired.
Rather then going for a script some systemd dependency integration seems plausible.

FYI ideas about an unlock service are around and probably just need some glue:
https://wiki.archlinux.org/index.php/ZFS#Unlock_at_boot_time
https://github.com/openzfs/zfs/issues/8750

What do Proxmox staff and lovers think with about wait dependencies with systemd.
Idea is to evaluate systemd service that deals with unlocking encrypted datasets.
What PVE systemd services could this should to unlock service declare as "Before="?

 

[Dunuin]

There is the zfs-mount-generator which should auto mount zfs datasets. I also wanted to use that but was to stupid to set it up. A tutorial in the wiki would be very nice. As far as I understand it can mount encrypted datasets but these need to use "raw" and not "passphrase" as encryption key. I wasn't able to find a way how to auto unlock my passphrase encrypted datasets.

 

[kraut.hosting]

@Dunuin Thanks for the hint but check the Shell script at /lib/systemd/system-generators/zfs-mount-generator
Sofar it doesn't integrate at all with Proxmox and also only set dependencies for the keyload service needs:

DefaultDependencies=no
Wants=${wants}
After=${wants}
${pathdep}

For minimal integration we would need to declare that PVE services needs keyload and import ran before.
Is just setups that put all Proxmox instance on encrypted datasets and force the requirement to unlock.
Manipulation the PVE systemd units is nogo as that would break on packages updates every time.
We should put "Before=" on the custom keyload service but need the best PVE service to put there.
Our best guesses are currently those ones that shouldn't interfere with cluster and firewall things:

/usr/lib/systemd/system/pve-container@.service
/usr/lib/systemd/system/pve-guests.service

Feedback from some ProxmoxPro desired if delaying those services should make autostart work then.

 

[kraut.hosting]

We did now the PoC of a simple ZFS decrypt service with wait dependencies on VMs/containers.
Tested only VMs with autoboot sofar but this setup serves the our use case of encrypt rpool/data.
If someone like to use single node Proxmox with ZFS encryption add this for encrypted dataset:
/etc/systemd/system/zfs-decrypt-pve.service

[Unit]
Description=Decrypt and mount PVE dataset rpool/data
Before=systemd-user-sessions.service
After=zfs-import.target
Before=pve-guests.service pve-container@.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/bash -c 'until (systemd-ask-password --id="zfs:rpool/data" "Encrypted ZFS password for PVE rpool/data" | zfs load-key rpool/data && zfs mount rpool/data); do echo "Try again!"; done'

After changing this systemd service change we must reload systemd and to have it available:

root@pve:~# systemctl daemon-reload

After reboot and login via console/SSH we just restart it and it asks for the passphrase:

root@pve:~# systemctl restart zfs-decrypt-pve
Encrypted ZFS password for PVE rpool/data *******************

It's a poor man's solution that we're having in testing now for encrypted backup servers.
Eecommend to upgrade Proxmox to have ZFS v0.8.4 that improves encryption performance.