Natively encrypted ZFS + additionally HDD with LUKS - cannot add HDD to existing pool.

[Proxomx]

Dear Community,

Thanks to your great help and tutorials I found here I managed to successfully natively encrypt ZFS running Proxmox (disc no. 1 NVMe), then add my storage (disc no.2 -> 12TB) with

parted /dev/sda mklabel gpt
parted -a opt /dev/sda mkpart primary ext4 0% 100%
mkfs.ext4 -L Storage /dev/sda1

and encrypted it with LUKS. I even managed to get everything working remotely through SSH with beardrop on boot + automatically loading HDD key.

The problem is I didn't think this whole concept through haha I got Wireguard and Plex LXC running but when came to implementing NFS or SMB in LXC (I have to yet decide, would also appreciate your ideas) I realize my rpool consist only of disc no.1 (NVMe with Proxmox and VT/CT on it). HDD storage is added in node (visible in discs and also as a "Storage" / type - directory).

I can access it without problems in shell but as the drive is in use / mounted I cannot add it to my existing pool with command

zpool add rpool /dev/sda

becasue I get an error

/dev/sda is in use and contains a unknown filesystem.

Also due to this error I cannot create new ZFS pool with this HDD (disc is not visible to mount).

My questions are->
1. Do I have to format again HDD, then add it to my existing pool and after that encrypt it with LUKS? If yes - what would be best steps (disc is right now decrypted, should I format it now / encrypt it first / doesn't matter because all data get erased nevertheless)?
2. Somehow umount HDD so I don't lose encryption and create new ZFS pool / add to existing one?
3. Leave it as it is - but how would you then proceed further from this point?
4. I also have additional 1TB SSD which I thought to make as a backup storage for VM / CT - should also add it to existing pool or create a new one (if yes then LVM or ZFS)?

I'm at the point I'm a little bit confused, need a little push to get things rolling!

Best regards,
Patrick

 

[Dunuin]

I can access it without problems in shell but as the drive is in use / mounted I cannot add it to my existing pool with command

zpool add rpool /dev/sda

becasue I get an error

/dev/sda is in use and contains a unknown filesystem.

Also due to this error I cannot create new ZFS pool with this HDD (disc is not visible to mount).

I don't get what you are trying and I think you don't understand ZFS.

So your got a ZFS pool using your NVMe and ZFS is using native encryption. That is fine.
Than you got a single HDD encrypted with LUKS using ext4 instead of ZFS. Thats fine too.
What won't work is adding your LUKS encrypted HDD to your existing ZFS pool. And in general it would be a very bad idea to add a HDD to a ZFS pool that is using a NVMe because both will be striped together, so your fast NVMe will be slowed down to the performance of your slow HDD. This will also lower reliability as you loose both SSD+HDD in case any of the two may die. So there is really no point to mix your SSD and HDD as a single pool.

My questions are->
1. Do I have to format again HDD, then add it to my existing pool and after that encrypt it with LUKS? If yes - what would be best steps (disc is right now decrypted, should I format it now / encrypt it first / doesn't matter because all data get erased nevertheless)?
2. Somehow umount HDD so I don't lose encryption and create new ZFS pool / add to existing one?

Why do you want ZFS at all for that HDD? If you need ZFS features use ZFS with native encryption instead of LUKS. If you don't need ZFS features use LUKS with etx4/xfs/LVM-Thin instead of ZFS for way less overhead, better performance and more usable capacity.

3. Leave it as it is - but how would you then proceed further from this point?

If you want to bring a SMB/NFS share into a unprivileged LXC use bind-mounts with user remapping. It doesn't matter if its on a native ZFS encrypted pool or a LUKS encrypted disk. You just work with mountpoints of filesystems that are stored on the unlocked disks.

4. I also have additional 1TB SSD which I thought to make as a backup storage for VM / CT - should also add it to existing pool or create a new one (if yes then LVM or ZFS)?

You should encrypt that SSD too. Wouldn't make sense to encrypt your VM storage if you then store copies of the VMs on an unencrypted disk. I would use LUKS with xfs for that in case you want to use zvdump.
If you want to setup a PBS, a unencrypted disk as a datastore should be fine, because PBS can encrypt your backups in software. I nthat case I would just use a normal ext4/xfs partition.

 

[Proxomx]

your fast NVMe will be slowed down to the performance of your slow HDD

OK, that's what I also read - not to mix drives in pool but at some point I saw everyone doing mirror ZFS with theirs HDDs and thought it would somehow work out...

If you don't need ZFS features use LUKS with etx4/xfs instead of ZFS for way less overhead, better performance and more usable capacity.

OK, thank you for clearing that out! I'm still new and have a moments where I get really confused at the time see everywhere ZFS

If you want to bring a SMB/NFS share into a unprivileged LXC use bind-mounts with user remapping.

OK - I found little comparison between NFS/SMB, bind mounts and cloudron but will try to implement your solution.

About last part with SSD - thank you again for suggestion! Maybe PBS will be better idea, already watched how to set it up.

@Dunuin Thank you again for helping me out, if that's not a problem and I would have next dumb/noob questions could I @ at you here, in this thread? I read a lot so hopefully I won't be bothering you (any)more / soon

 

[Dunuin]

OK, that's what I also read - not to mix drives in pool but at some point I saw everyone doing mirror ZFS with theirs HDDs and thought it would somehow work out...

But in case of a mirror mixing a for example 1TB SSD with a 12TB HDD would result in a SSD with the performance of a slow HDD and only 1TB of usable storage, so you would waste the 11TB of that HDD. So thats even worse, because you basically get the same as mirroring two slow 1TB HDDs. You could prevent wasting the 11TB by just using a 1TB partition of that HDD for the mirror and use another 11TB partition for something else, but in that case you NVMe is still slowed down to HDD speeds so VMs/LXCs would get really slow.

And you don't need ZFS for mirroring. I for example got a mdadm software raid1, ontop of that LUKS and ontop of that LVM. So a encrypted raid1 without any ZFS. I used it that way to install my PVE on that because back than it wasn't possible to boot from an encrypted ZFS pool.
But if you want mirroring/raid1 (which is always a good idea if you got the money for it) don't mix SSD/HDDs or disks with different capacities or protocols and instead always buy similar disks in pairs.

So if you want availability get 2 similar SSDs, use ZFS to mirror them and use them for boot/system and as a VM/LXC storage. If you want some additional cold storage get 2 similar HDDs, use ZFS to mirror them and then setup some kind of NAS with them so your guests can access them using NFS/SMB shares or bind-mounts. And raid never replaces a backup, so you want atleast a third even bigger HDD to store your backups on so you always got a copy of all the important data stored on that SSD and HDD pools.

If you don't need availability or bit rot protection forget about mirroring/raid and just use single LUKS encrypted disks with LVM/LVM-Thin ontop of it. And another disk for backups ofcause.

About last part with SSD - thank you again for suggestion! Maybe PBS will be better idea, already watched how to set it up.

Jep, I also would prefer that.

@Dunuin Thank you again for helping me out, if that's not a problem and I would have next dumb/noob questions could I @ at you here, in this thread? I read a lot so hopefully I won't be bothering you (any)more / soon

You, can but I probalby read it anyway