Encrypt ZFS pool

[axrusar]

I setup a small Proxmox server with 2 identical 240gb SSDsand ZFS raid 1.

Now i would like to add two 2TB spinner drives for bulk storage, also on raid 1 and zfs, but i would like to encrypt it.
I found many tutorials setting up a ZFS pool in proxmox with the new drives, but i do not see an option to encrypt.

I want to protect the physical disks from being read externally in the event the machine ever gets stolen.
Sorry for the newbie question

 

[fireon]

It's too easy, really!

Create one encrypted dataset under a pool, like this:

zfs create rpool/safe -o encryption=on -o  keyformat=passphrase

And all datasets created after safe like this:

rpool/safe/blabla

are automaticly encrypted. So vor example create an dataset with the name vmdata under safe

zfs create rpool/safe/vmdata

This dataset you add as storage in the webinterface. Now you can create your VM's and CT's.

Important: If you add an encrypted dataset to an rpool, you have to use the Proxmoxboottool, if not, your server is not bootable anymore. Also an migration or replication from the Proxmoxwebinterface is not available anymore, because encrypted datasets/VM's are not supported via the webinterface yet.

After Proxmox reboot, enter the command "zfs mount -l -a" to mount all encrypted datasets.

https://deepdoc.at/dokuwiki/doku.php?id=server_und_serverdienste:linux_zfs_verschluesselung

 

[Dunuin]

Important: If you add an encrypted dataset to an rpool, you have to use the Proxmoxboottool, if not, your server is not bootable anymore. Also an migration or replication from the Proxmoxwebinterface is not available anymore, because encrypted datasets/VM's are not supported via the webinterface yet.

Do you know if even an offline migration won't work when both zfs pools are unlocked? I was thinking about a cluster but all my pools are encrypted, so a cluster won't make sense at all when even offline migration won't work. I thought just replication won't work with encrypted pools when using HA and live migration.

 

[fireon]

Do you know if even an offline migration won't work when both zfs pools are unlocked? I was thinking about a cluster but all my pools are encrypted, so a cluster won't make sense at all when even offline migration won't work. I thought just replication won't work with encrypted pools when using HA and live migration.

With CLI yes. There is an special option for RAW transfer. I will have a look at evening.

 

[Dunuin]

I also asked in another thread and fabian confirmed that offline migrations would rely on the ZFS replication which isn't supporting ZFS native encryption.
But unencrypten ZFS ontop of sohething like LUKS would work.

Would be great if that raw option would work.

 

[axrusar]

I need to order a few drives. Once i get them you guys may guide me a bit to setup the encrypted pool.
I do not intend to encrypt the main rpool that proxmox runs on, so do i still need to setup all the Proxmoxboottool part?

 

[Dunuin]

I do not intend to encrypt the main rpool that proxmox runs on, so do i still need to setup all the Proxmoxboottool part?

As far as I undestand systemd (so the proxmox-boot-tool) boot is just required when you store encrypted data on your boot pool as grub couldn't boot from that. If you don`t boot from that pool you should could gnore that part.

 

[fireon]

As far as I undestand systemd (so the proxmox-boot-tool) boot is just required when you store encrypted data on your boot pool as grub couldn't boot from that. If you don`t boot from that pool you should could gnore that part.

If you have this output:

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
F6A1-98F4 is configured with: uefi (versions: 5.4.34-1-pve, 5.4.78-2-pve), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve, 5.15.35-1-pve, 5.15.35-2-pve)
F6A1-FD1A is configured with: uefi (versions: 5.4.34-1-pve, 5.4.78-2-pve), grub (versions: 5.13.19-6-pve, 5.15.30-2-pve, 5.15.35-1-pve, 5.15.35-2-pve)

It is ok to create some ZFS encryption on rpool.

If you see this output:

proxmox-boot-tool status                            
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.

it is a bad idea. Then have to switch to Proxmox Boot Tool: https://pve.proxmox.com/wiki/ZFS:_Switch_Legacy-Boot_to_Proxmox_Boot_Tool

https://forum.proxmox.com/threads/v...-auf-rpool-macht-das-system-unbootable.96374/

 

[fireon]

I also asked in another thread and fabian confirmed that offline migrations would rely on the ZFS replication which isn't supporting ZFS native encryption.
But unencrypten ZFS ontop of sohething like LUKS would work.

Would be great if that raw option would work.

Here are my topic: https://forum.proxmox.com/threads/z...rschlüsselten-datasets-vms.96510/#post-418243

And here are the Bugreport: https://bugzilla.proxmox.com/show_bug.cgi?id=2350

 

[axrusar]

It's too easy, really!

Create one encrypted dataset under a pool, like this:

zfs create rpool/safe -o encryption=on -o  keyformat=passphrase

And all datasets created after safe like this:

rpool/safe/blabla

are automaticly encrypted. So vor example create an dataset with the name vmdata under safe

zfs create rpool/safe/vmdata

This dataset you add as storage in the webinterface. Now you can create your VM's and CT's.

Ok i am ready now.
Here is my new pool

So i tried this:

 create rpoolBULK/crypt -o encryption=on -o  keyformat=passphrase

No hiccups. i set the passphrase and all good.
Now... where do i see it? how do i use it?
For example i had a VM already setup. If i add another hard disk to it, i can now pick the rpoolBULK. But i do not see that specific "crypt" dataset as an option.
If i just pick the drive as i did, that means the data is encrypted there or i have to point it to the crypt dataset somehow?
I am absolutely lost and new to these "datasets" and the way zfs works, excuse my ignorance.

 

[Dunuin]

ZFS will inherit attibutes to childs. So if you encrypt a dataset every dataset and zvol ontop of it will by default also get the encryption inherited.

To actually use the encryption you need to add a new storage of type ZFS pointing to that encrypted dataset and move/create your virtual disks there.

 

[axrusar]

ZFS will inherit attibutes to childs. So if you encrypt a dataset every dataset and zvol ontop of it will by default also get the encryption inherited.

To actually use the encryption you need to add a new storage of type ZFS pointing to that encrypted dataset and move/create your virtual disks there.

Thank you Dunuin for your guidance so far.. i think i start to get the idea..Im still trying to figure it out.
I am reading the ProxMox manual, section 7 Proxmox VE Storage.

And now i just deleted a bunch of unnecessary questions after i found section 3.8.9. Encrypted ZFS Datasets

i run this:

pvesm add zfspool encrypted_zfs -pool rpoolBULK/crypt

And the new storage option popped up in the GUI
I think my brain finally clicked and got the idea of how these block storage systems work. I see you can have as many as you want with any options, and the space is virtually allocated so i can have 50 different datasets of 2TB if i wanted and the space is dynamically taken right?

Thank you to the other guys that gave some input too.

 

[Dunuin]

I think my brain finally clicked and got the idea of how these block storage systems work. I see you can have as many as you want with any options, and the space is virtually allocated so i can have 50 different datasets of 2TB if i wanted and the space is dynamically taken right?

Jup. The magic of thin-provisionuling.

 

[axrusar]

Important: If you add an encrypted dataset to an rpool, you have to use the Proxmoxboottool, if not, your server is not bootable anymore. Also an migration or replication from the Proxmoxwebinterface is not available anymore, because encrypted datasets/VM's are not supported via the webinterface yet.

Ok sorry i am coming back to this part. Just want to make sure i don't mess all the work done already.
I got

240gb SSD drives:
rpool

2tb Slow spinner drives:
rpooBULK
rpoolBULK/crypt

I want to use the big 2tb (slow) pool to hold data inside the encrypted dataset shared among different VMs.
But i want the VMs to perform fast under the SSDs.

I prefer to have the VMs also encrypted so i need to create a new dataset under rpool (rpool/crypt)
So just by adding that dataset, i DO have to install the Prozmoxboot tool? or the actual proxmox installation is not affected by the new dataset?
The proxmox installation itself can reside un-encrypted in rpool, i do not care about encrypting the whole thing.
Thank you

 

[Dunuin]

As far as I understand the german link (https://deepdoc.at/dokuwiki/doku.php?id=server_und_serverdienste:linux_zfs_verschluesselung) your PVE host won't boot anymore, in case it is using grub to boot and not systemd, no matter if you encrypt the whole rpool pool or just a dataset ontop of it.

So in case you want to encrypt your rpool and it is using grub you would need to...
1.) replace your hardware if it only got BIOS and no UEFI or
2.) in case it got UEFI you would need to disable CSM in UEFI
...and then install PVE again, where it then should use systemd-boot with the proxmox-boot-tool instead of grub.

 

[axrusar]

As far as I understand the german link (https://deepdoc.at/dokuwiki/doku.php?id=server_und_serverdienste:linux_zfs_verschluesselung) your PVE host won't boot anymore, in case it is using grub to boot and not systemd, no matter if you encrypt the whole rpool pool or just a dataset ontop of it.

So in case you want to encrypt your rpool and it is using grub you would need to...
1.) replace your hardware if it only got BIOS and no UEFI or
2.) in case it got UEFI you would need to disable CSM in UEFI
...and then install PVE again, where it then should use systemd-boot with the proxmox-boot-tool instead of grub.

Alright. No, too much trouble i believe.
I am totally fine encrypting the file systems or home folders as needed inside each of the 3 or 4 VMs i will run total.
So i will just attach the encrypted pool as a secondary hard drive to the VMs and thats it.

Final question here..
So if i use my crypt dataset as a bulk storage place, can i safely attach it and share it among different OSs inside VMs?
Would taking snapshots of the VMs copy the way the data in the crypt dataset was right at that time and create a conflict of data with the others?
Thanks a lot.

 

[axrusar]

Ok i setup my first VM server.
Now i am trying to attach the encrypted pool i previously created as a second hard disk.
The disk shows "available" in the proxmox gui...
But i get this:

So where should i load/provide the key now? (i assume the key is the passphrase i used to encrypt it?) Sorry for so many questions guys..
i could not figure this one out with the manual.

EDIT:..
Ok got it... i just tried mounting zfs mount -l rpoolBULK/crypt
I am able to add the second drive now.

So i guess if i reboot the ProxMox server, the mount will be gone.
Do i just add it in fstab?, and if the passphrase needs to be left somewhere in plain text, that defeats the whole purpose
of everything done so far, looking how to get around this now.

 

[fireon]

So i guess if i reboot the ProxMox server, the mount will be gone.
Do i just add it in fstab?, and if the passphrase needs to be left somewhere in plain text, that defeats the whole purpose
of everything done so far, looking how to get around this now.

After an reboot you have to enter the password manually. You can do this with this command:

zfs mount -l -a

 

[axrusar]

Guys thank you for the help provided and awesome guidance.

Unfortunately i feel pretty pissed off, not with any of you, but with the fact that i could not get to mount the encrypted dataset automatically on startup. I find it hard to believe such a robust piece of software does not provide such a basic feature via the GUI, and so far i could not find any solution via the terminal.
I wasted many days setting up my home server with ProxMox but i will have to pass and use something else.
I found TrueNAS to have this functionality when setting up a dataset, it lets you save the keys and not have to go into the terminal when you are a continent away from home and the power goes off..

Example:

No passphrases needed, all automatic and you get the keys if anything goes wrong..

[/rant=OFF]

 

[Dunuin]

Jup, would be very nice if encryption could be added to PVE. Right now it works but just CLI and you will have to setup everything yourself which isn't easy.
But TrueNAS isn't perfect either. I'm for example missing the full system encryption there that I can use with PVE.

But if you setup PVE right, it works great. After rebooting the full system is locked. I then just need to open a SSH session (no typing in of the password is required as Keepass stores the RSA key and adds it to pagean as soon as I unlock my password safe) and it will ask me for the passphrase. I then copy and paste it from my Keepass password safe into the ssh client and the root filesystem will be unlocked and PVE boots. On the root filesystem I got the keys for all the datasets so after PVE has finished booting and before starting the guests a script I made will unlock all my datasets and mount them.
So all I have to do is unlock my password safe, double click on Putty, copy and paste my master passphrase and thats it. And as no physical access is needed with SSH, I can unlock my PVE from anywhere, as long I got a internet connection and a VPN client. Takes me way longer to unlock all the datasets manually in TrueNAS.