Proxmox Full Disk Encryption with ZFS Raid1 on LUKS | A couple last questions

[Anotheruser]

So for the past couple weeks i have been looking into using full disk encryption on all my hosts.
I literally went through all sorts of different setups (got like 20 test VMs by now) and guides...

What i am still not sure about is the way for the final proxmox setup,

is it better to
Option 1: just live boot debian do all the storage setup there and then install proxmox as an application on that debian or
Option 2: first install proxmox on that server normally via the proxmox iso (just as simple zfs raid1) on two other temporary disks,
then use the debian live boot to just setup the luks, boot and efi sections on the real disks and
after that finally just move the proxmox partitions from the temporary disks (via zfs export / import) on top of the luks mapped devices?
(btw the installation on the temp drives is done on the actual server thats later going to run that proxmox install to make sure all the devices & stuff like network configs is set correctly for that particular host)

I am currently leaning more to option 2 but wanted to ask what your opinion on that is.

What i figured out during testing so far
The base setup that is identical for both options (debian live boot, creating the partitions & luks setup) is quite straight forward and doesnt cause any major headaches.
Where the issues with option 1 started was the debian installation from the live boot on the disks just created disks. Numerous errors during install and while the system booted from the disks mostly fine, the entire debian install was completely weird, a lot of different settings where just very different from even a normal debian install via the guided setup, i also pretty bad stability issues on that install. (i dont know how i messed up )
Also since proxmox does change quite a lot of things when installed via the official proxmox iso compared to a installation thats done manually on top of debian i would prefer to have everything like when installed via the official iso.

So yeah do you think it is a good idea to just move the zfs volumes from a proxmox iso install on top the luks volumes?

 

[Dunuin]

Option 1: just live boot debian do all the storage setup there and then install proxmox as an application on that debian or

Problem with the Debian installer is, that it supports LUKS but not ZFS.
PVE installer supports ZFS but not LUKS.
So whatever you do, you will have to heavily modify your installation via CLI from a Live linux with ZFS support.
My guess would be that option 2 is easier to achieve.

Didn't tried LUKS + ZFS yet, but converting an unencrypted PVE installation to one using ZFS native encryption is well documented and is working fine for years. Biggest problem with ZFS native encryption is still, that no migration/replication will work. You don't have that problem with ZFS on LUKS. ZFS on LUKS is a bit more exotic but is working too (one of the devs once mentioned using this).

So yeah do you think it is a good idea to just move the zfs volumes from a proxmox iso install on top the luks volumes?

You an make use of "zfs send ... | zfs recv ... " to move the root dataset between pools.

 

[Anotheruser]

Problem with the Debian installer is, that it supports LUKS but not ZFS.
PVE installer supports ZFS but not LUKS.
So whatever you do, you will have to heavily modify your installation via CLI from a Live linux with ZFS support.
My guess would be that option 2 is easier to achieve.

Didn't tried LUKS + ZFS yet, but converting an unencrypted PVE installation to one using ZFS native encryption is well documented and is working fine for years. Biggest problem with ZFS native encryption is still, that no migration/replication will work. You don't have that problem with ZFS on LUKS. ZFS on LUKS is a bit more exotic but is working too (one of the devs once mentioned using this).


You an make use of "zfs send ... | zfs recv ... " to move the root dataset between pools.

thanks, yeah i have done a couple installs with zfs native encryption already and i am fully aware that this is heavily customized and by no way officially supported
the main drawbacks of zfs native encryption are in my opinion that migration within a cluster doesnt work anymore (a must have for me since over 15 nodes in that cluster and stuff gets constantly moved) and that i leaks all the vm ids (bc it creates a separate volume for each one) including how much disk space is used by each vm among a bunch of other things.

Actually now thinking about that shouldnt even be a problem for my setup since i am using two small disks as dedicated proxmox disks without any vms... that proxmox pool could run zfs native encryption fine and only the vm pools would be zfs on luks which also shouldnt be a problem and a pretty easy setup since its not the root / boot pool.

My last test install was basically debian live boot, installed cryptsetup, installed zfs, then created boot, efi and a luks partition on each drive, then created a zfs mirror pool with the two boot partitions and another zfs mirror using the luks mapped devices.
Like mentioned so far worked flawlessly. Where it fell apart was when i tried to install debian

But wow now that i think about it it might actually be better to just run zfs native on the proxmox disks and then zfs on luks only on the vm pools...

Anyway i learned a bunch of interesting stuff about filesystems, zfs & luks during the past weeks

 

[Nemesiz]

My story of LUKS and ZFS

For one server I needed to encrypt as much as possible with auto password insert at boot.

I did it in using Proxmox 5.0 at that time and from that moment system still running up to this day.

Partitions:
for system raid disks I splitted into unencrypted boot partitions and LUKS encrypted partitions for root and data. This was done using Debian live cd and than converted to Proxmox.

Boot loaded:
GRUB at that time. Now you can use proxmox-boot-tool tool to update all boot partitions in different disks automatically. But I stick with GRUB for that server just not to allow unintended boot meniu editing.

Password autoloader:
I use Mandos ( https://www.recompile.se/mandos ) to load password at boot time and unlock LUKS. I had to do some changes over time with Proxmox upgrades to keep it work, but it do his job.



As of ZFS native encryption I have not much experience (in crash situations). I`m happy that they added it but from ZFS #irc talks not everyone agree that it is stable. I use it in backups server for now only.

 

[Stereoscope]

I recently started researching encryption with Proxmox. I started researching Proxmox not too long before that. So at present I have more questions than answers so excuse the question if it's totally off base.

Why is everyone using Debian as the base install to install Proxmox over if it doesn't support ZFS? Why not install Ubuntu with ZFS and Proxmox on top of that? If you install with Ubuntu you can even use the default OpenZFS encryption? No?

 

[Dunuin]

Because PVE is based on Debian userland and Debian is the only supported linux distribution to install PVE on top of.

 

[Stereoscope]

Ah I see, thanks.

 

[Anotheruser]

Ah I see, thanks.

Btw a little continuation from my journey with that and also which steps i tried.
https://forum.proxmox.com/threads/r...y-delaying-the-zfs-import-during-boot.140285/

1. You basically have three encryption setups to choose from
1.1. ZFS Native encryption - easiest and tons of guids for it, but you wont be able to migrate VMs between host and it also leaks certain data.
1.2. Luks with LVM - didnt look into it much but this would be the way if you dont want to use zfs for what ever reason
1.3. ZFS ontop Luks - in my opinion the absolute best when you get it running - barely leaks any data, migration still works but also the most complex to setup

2. Then you have to choose between two ways how to set it up:
2.1. Install lrom the ground up from a base debian machine - huge amount of work, you will need to configure everything manually and probably will never get to a reliability level comparable to the official iso, i tried this for days and gave up on that
2.2. Install from the proxmox iso and move all the files over to encrypted disks afterwards. - still a ton of work but doable and it pretty much has all the settings from a normal official iso based install

I personally chose 1.3 and 2.2 bc it seems to be the easiest way to get all the security and features i want, but you will definitely have to know a thing or two about linux administration in general bc it is still quite complex.

I mostly got it working over way 2.2 with the only issue it being stuck at boot in certain situations, which is probably caused by a small miss configuration in the boot or efi section (i havent found it tho so far see the other post i linked above)

Anyway it would be incredible if someone knew the solution to that or wants to help in general with creating a guide for those setups.
I have seen the encryption questions come up quite a few times in the past and it would be great if there would be a start to finish guide for zfs on luks (like there is for zfs native) since zfs native is still not an option for a lot of setups due to the restrictions of migrating vms.
Feel free to post here or dm me here if you want to help with that guide (or found one)

 

[Dunuin]

I will probably try 1.3 + 2.2 with and virtualized PVE once I finished re-doing my network.

Did you got it working that LUKS will be unlocked before ZFS pools get imported? As far as I see it should work like this for my needs:

1.) Installing unencrypted PVE from PVE ISO as ZFS mirror and tell it to only use 33GB of the disk so it creates 1MB lecagy partition, 1GB ESP, 32GB rpool just for the OS. Remaining disk space unallocated for later use.
2.) manually create 2 new partition. One 4th 8GB partition for swap. Remaining space for 5th partitions for another ZFS pool called "tmppool" or later "dpool".
3.) create an mdadm raid0 using the 8GB swap partitions. Encrypt that mdadm raid array with LUKS. Create swap on top of that LUKS.
4.) boot ZFS capable Live Linux and import rpool
5.) create "tmppool" ZFS Pool on top of the big 5th partition. Use "zfs send | zfs recv" to copy rpool/ROOT/pve-1 to tmppool.
6.) destroy rpool and encrypt both 3rd partitions with LUKS
7.) create new rpool on top of the now LUKS encrypted 3rd partitions
8.) copy tmppool/pve-1 back to rpool/ROOT/pve-1
9.) destroy tmppool
10.) encrypt 5th partitions via LUKS
11.) create new pool "dpool" on top of LUKS for storing VMs and data
12.) chroot into rpool
13.) add cryptsetup support and dropbear-initramfs to initramfs
14.) fix stuff to get PVE working again. Maybe writing new bootloaders, rebuilding initramfs, editing systemd services so unlocking LUKS has to be done before importing ZFS Pools or mounting swap and so on
15.) boot into PVE and set up dpool as storage for VMs and data
16.) create some scripts to unlock other disks via keyfiles stored on rpool (so first unlocking the disks via LUKS and then importing the ZFS pool on them)

 

[Anotheruser]

thanks for your detailed steps,
i did it using the re-silvering / replace command instead of messing around with a temp partition (i do not store any vm disk on the root disks)
The thing i am currently stuck with (details described in my other post https://forum.proxmox.com/threads/r...y-delaying-the-zfs-import-during-boot.140285/)

Is that zfs tries to import the pool before the luks devices are unlocked therefore causing a failed import but the luks mapped devices generally work. If i just run a simple zfs clear command after boot everything returns to normal.
Any ideas?

 

[Dunuin]

You could try this: https://serverfault.com/a/1035667

 

[Anotheruser]

You could try this: https://serverfault.com/a/1035667

I just tried it but doesn't change anything, zfs still tries to import the pool before luks is fully unlocked / mapped.
Do i need to add anything particular that the service gets executed before all the other thing?

 

[Anotheruser]

After countless hours, it works finally!

Super happy

Huge thanks to everyone that helped.

What i ended up doing was just switching from systemd-boot to grub.
I just reinstalled it with secure boot enabled so it automatically defaulted to grub.

The zfs import delay thing turned out to be unnecessary!
You just need to add initramfs to the /etc/crypttab file and thats it, then it automatically initializes before zfs without any additional changes or modifications to the systemd services.
Your crypttab should look like this Example

luks1 UUID=6785675567b-37897-4876b-b0188123463242344 none luks,discard,initramfs

one line for each disk

 

[Doorstep2784]

Hello everyone,

I'm one of the people taken by surprise, that a fully encrypted ZFS cluster won't allow migration of VMs. I do not have special ZFS needs (so far), but was tempted by the well documented full host encryption that comes with it.

Reading about it, a solution doesn't seem trivial. However, one way seems to do the trick quite quickly: Putting the VMs intended for migration on a Directory type storage, which can be located under the encrypted ZFS pool.

What I don't grasp yet is: What disadvantages come with this solution? The disks seem to be file based now (.qcow2) - other than that it seems to work fine. Will it be much slower? Would there be way better alternatives if a file-based storage is enough?

I'm just starting the setup and experimenting with it, so I would love to hear better now than later if that's a complete no-go. Thanks a lot!

 

[esi_y]

I'm one of the people taken by surprise, that a fully encrypted ZFS cluster won't allow migration of VMs. I do not have special ZFS needs (so far), but was tempted by the well documented full host encryption that comes with it.

Reading about it, a solution doesn't seem trivial. However, one way seems to do the trick quite quickly: Putting the VMs intended for migration on a Directory type storage, which can be located under the encrypted ZFS pool.

Isn't it better to simply use SED or LUKS?

What I don't grasp yet is: What disadvantages come with this solution?

No replication possible.

The disks seem to be file based now (.qcow2) - other than that it seems to work fine. Will it be much slower? Would there be way better alternatives if a file-based storage is enough?

I'm just starting the setup and experimenting with it, so I would love to hear better now than later if that's a complete no-go. Thanks a lot!

You can benchmark it how much overhead it actually adds.

 

[Doorstep2784]

Isn't it better to simply use SED or LUKS?

Well, that is part of the question - is it? It seems way more complex to setup LUKS (mainly because you need to start from plain Debian) than to "just" use the ZFS encryption - but maybe LUKS is better if one doesn't specifically need ZFS?

No replication possible.

Fair point. If one relies on other means of backups as well (e.g. PBS), I - without much knowledge - would put this point under "convenience" that has to be weight in versus the convenience of being able to use migration.

You can benchmark it how much overhead it actually adds.

Highly interested in that - are there any recommended tools / commands to do this?

 

[esi_y]

Well, that is part of the question - is it? It seems way more complex to setup LUKS (mainly because you need to start from plain Debian) than to "just" use the ZFS encryption - but maybe LUKS is better if one doesn't specifically need ZFS?

Ok I might be biased, but any data-only encryption, no metadata, is almost pointless. For lots of use cases, encryption at rest means ticking a compliance item, that cannot be data only. Besides that, either one objectively needs encryption or does not. Whenever possible, that should be SED nowadays, LUKS made sense for the commodity spinning drives and was a nice exercise. I guess what I meant to say was, why not FDE, if at all.

There actually was a rather elaborate (close to tutorial [1]) post on exactly that setup from scratch. For me the ISO (especially when using ZFS since it makes it for root too and the hocus pocus with shoving /boot into EFI) is so bad it even makes no sense to install other than on top of Debian when using ZFS.

Fair point. If one relies on other means of backups as well (e.g. PBS), I - without much knowledge - would put this point under "convenience" that has to be weight in versus the convenience of being able to use migration.

What's the whole point of ZFS without replication (not just within PVE)? Snapshots? I understand it might come as provocative, but really ... consider with solutions like PBS it's not even taking advantage of the native send/receive, but wasting CPU cycles with filesystem agnostic solution. [2]

Highly interested in that - are there any recommended tools / commands to do this?

Probably the usual fio batches with obscure switches and highly subjective results. I believe @Dunuin knows these the best actually from past posts.

[1] https://forum.proxmox.com/threads/proxmox-8-luks-encryption-question.137150/#post-609786

[2] https://forum.proxmox.com/threads/actual-zfs-send-receive-backups.136222/

 

[Doorstep2784]

Disclaimer: The following paragraphs might indicate that I do not know enough about these topics and I don't want to bore anyone with discussions probably held a thousand times. However, since you took your time to answer me in-depth, I would also feel bad to not reply at all:

Thanks a lot for this input! I don't have the expertise to judge the pros and cons of a certain way of encrypting - but in times where even western governments advocate for the end of end-to-end encryption, I do have a hard time trusting hardware and therefore vendor-encryption (e.g. SED) more than OS software encryption.

That being said, I'm also not a big believer of a black and white, all-or-nothing approach: Data-only encryption might not work against a government entity, but someone eager enough just needs a 5$ wrench anyways. However, as a home-user, any kind of data-encryption should be good enough for most cases - I do not want to tempt anyone to take my hard drives for a quick readout just because they one day remember me being the guy that "mentioned crypto once".

So yes, I would feel better with encrypted drives, even if a sophisticated entity could still break in - stealing hardware is just too easy. And that's the point where ZFS seems to be the easiest solution. But since I do have two nodes to run a dedicated PBS, I thought of using migration / replication as well. LUKS might solve the former, but it seems that replication only works with ZFS - therefore, I probably won't switch my encryption method and just rely on PBS to "migrate" via the backup if in need. Thanks for the input!

 

[esi_y]

I do have a hard time trusting hardware and therefore vendor-encryption (e.g. SED) more than OS software encryption.

LUKS would be best bet, the issue for most is performance, nowadays at least you have AES-NI on every CPU, but still with modern NVMe drives you will soon start hitting a ceiling when it comes to performance. Other than that, LUKS is perfectly transparent solution.

However, as a home-user, any kind of data-encryption should be good enough

You can't disregard SED and then mention this in one and the same rationale. Talking of which, what is your CPE at home? Often it's the likes of Huawei gear and most people do not run encrypted tunnels out of their ISP's network either, they trust the firmware is e.g. custom by the ISP and that's good enough.

stealing hardware is just too easy

Actually, I never used any spinning drives with anything other than LUKS if for nothing else than in case of need to RMA them arises and they are "not spinning" but need to be shipped back.

And that's the point where ZFS seems to be the easiest solution.

The encryption part of OpenZFS came in v0.8 that's around 2020, the feature ... well, have a look for yourself [1] and consider how mature it would be as opposed to LUKS which has been around since almost 20 years by now.

But since I do have two nodes to run a dedicated PBS, I thought of using migration / replication as well.

There you go.

LUKS might solve the former, but it seems that replication only works with ZFS - therefore,

ZFS on LUKS?

I probably won't switch my encryption method and just rely on PBS to "migrate" via the backup if in need. Thanks for the input!

That's fine too. Cheers!

[1] https://github.com/openzfs/zfs/pull/5769