Newbie: setting up Intel NUC with PVE and Ubuntu

[gctwnl]

I have been using the Unixy underlayer of macOS (Darwin) for many years and I have been running a server based on macOS for the last 20 years. So, I am somewhat technical. I am moving away from this (macOS is no longer reliable enough for this). I am running:

• nsd/unbound (in a complex setup with split-DNS and two parallel unbounds, one forwarding, one non-forwarding (for rspamd))
• Mail server (postfix, dovecot (nonstandard: apns-capable) , rspamd, etc.)
• Nginx (web server, and SSL termination for MinIO)
• 6 MinIO services on localhost, forwarded via nginx. Each MinIo service supports Duplicati for an external user

The old setup has an internal disk that is one half boot+data disk and one half TimeMachine backup (which runs permanently if you mount the thing during boot via /etc/fstab). Next to that, there is a backup system running to an externally connected TB3 SSD as well as to offsite storage. The external users use the MinIO services and these store their data on on external mirror RAID.

I've decided to move to an Intel NUC (10i5/1TB/16GB (oversized, yes)) with Proxmox 2.7 and Ubuntu 22.4.1LTS VM(s) for the new server. I've secured the NUC hardware (boot password, SSD hardware encrypted), installed PVE and installed one Ubuntu Server VM to work as the host for nginx/MinIO.

Virtualisation is new for me. Ubuntu is new for me every command line command is different almost). I am in a steep learning curve missing even some basic things. I am at this stage wondering about the following and could use some tips:

I would like to have the external USB3 RAID to have encrypted data (in case the drive is stolen — all my data is encrypted at rest to protect against data theft when I suffer physical theft). That either means I put only encrypted data (such as backups) there, or I have a way to have either PVE or Ubuntu encrypt data on that drive. Is the latter an option? Because if it is, I could host my mail/web live data there and have it on reliable storage. If not, the live mail/web data should reside on the internal LVM storage (900GB). So:

1. Can I have PVE or Ubuntu encrypt the data on external storage (like that USB3 hardware RAID)? How? Tutorial somewhere?
2. How do I give my Ubuntu a data mount from the local lvm so I can put my MinIO data there? I am getting lost in the PVE manual and need to know what steps to take to create a volume in local-lvm and mount that on Ubuntu during boot.

 

[Dunuin]

1. Can I have PVE or Ubuntu encrypt the data on external storage (like that USB3 hardware RAID)? How? Tutorial somewhere?

Ubuntu support encrypted partitions and even full system encryption using LUKS. Even through the ubuntu installer.
PVE doesn`t support encryption but you could use it with LUKS or ZFS encryption when setting that up manually. So an encrypted VM/LXC virtual disks storage is also an option. Even full system encryption with PVE is possible, but thats really not easy and quite hacky to setup.

1. How do I give my Ubuntu a data mount from the local lvm so I can put my MinIO data there? I am getting lost in the PVE manual and need to know what steps to take to create a volume in local-lvm and mount that on Ubuntu during boot.

"local-lvm" is a LVM-Thin pool and can only store virtual disks, so block devices in form of LVs. So you can only add another virtual disk to your VM, select local-lvm as storage and then your ubuntu VM should see a new unpartitioned virtual disk that you can partition and format to your needs to store stuff on it.

 

[gctwnl]

Ubuntu support encrypted partitions and even full system encryption using LUKS. Even through the ubuntu installer.
PVE doesn`t support encryption but you could use it with LUKS or ZFS encryption when setting that up manually. So an encrypted VM/LXC virtual disks storage is also an option. Even full system encryption with PVE is possible, but thats really not easy and quite hacky to setup.

"local-lvm" is a LVM-Thin pool and can only store virtual disks, so block devices in form of LVs. So you can only add another virtual disk to your VM, select local-lvm as storage and then your ubuntu VM should see a new unpartitioned virtual disk that you can partition and format to your needs to store stuff on it.

Thank you.

I do not need to encrypt local storage of my NUC (as it is encrypted SSD hardware) but I am only interested in using the external USB3 RAID and encrypt everything on that. I guess in theory I can do that either from (the Debian of) PVE (and then use PVE to define volumes on that and hand these to the Ubuntu client OS who sees it as unencrypted. My guess is that if I encrypt from PVE Debian host OS, I am very flexible in connecting that storage to client VMs and I do not have to manage encryption in the client OS. So my first option to investigate would be to go into PVE's Debian, attach the external USB and use LUKS(?) to encrypt the external drive? Is that possible?

Are there performance got-you's here?

 

[Dunuin]

Thank you.

I do not need to encrypt local storage of my NUC (as it is encrypted SSD hardware) but I am only interested in using the external USB3 RAID and encrypt everything on that. I guess in theory I can do that either from (the Debian of) PVE (and then use PVE to define volumes on that and hand these to the Ubuntu client OS who sees it as unencrypted. My guess is that if I encrypt from PVE Debian host OS, I am very flexible in connecting that storage to client VMs and I do not have to manage encryption in the client OS. So my first option to investigate would be to go into PVE's Debian, attach the external USB and use LUKS(?) to encrypt the external drive? Is that possible?

Jup. You didn't told us what raid the USB enclosure is using. If its HW raid and it shows the whole disk array as a single disk you could use LUKS to encrypt it. In that case ZFS wouldn't be a good choice as ZFS shouldn't be used on HW raid. But if that enclosure isn't using raid and will show you the individual drives it would be a good idea to use ZFS with its native encryption as a software raid so you got the better data integrity and additional features.

Are there performance got-you's here?

Usually the encryption algorithm itself isn`t the problem as all CPUs from the last 13 years or so should support the AES-NI instruction set, so the AES encryption is hardware accelerated and much faster than your storage could handle it. But your storage performance could degrade in case you take the encryption serious. Because then TRIM/discard isn't allowed because the holes would make it easiert to break the encryption because the attacker will know wht is data and what free space. And without TRIM/discard SMR HDDs and SSDs can't use the free space to optimize the performance so the disks will run as slow as a disk will do when completely filled (so no free space that could be used as CMR or SLC cache and no free space to move data around or be defragmented so it could be read/written faster later).

 

[Dunuin]

Also don't forget to encrypt your hosts and guests swap so you are not leaking the encrypted data, by caching it in RAM and then swapping that RAM to unencrypted swap partition.

 

[gctwnl]

Jup. You didn't told us what raid the USB enclosure is using. If its HW raid and it shows the whole disk array as a single disk you could use LUKS to encrypt it. In that case ZFS wouldn't be a good choice as ZFS shouldn't be used on HW raid. But if that enclosure isn't using raid and will show you the individual drives it would be a good idea to use ZFS with its native encryption as a software raid so you got the better data integrity and additional features.

Usually the encryption algorithm itself isn`t the problem as all CPUs from the last 13 years or so should support the AES-NI instruction set, so the AES encryption is hardware accelerated and much faster than your store could handle it. But your storage performance could degrade in case you take the encryption serious. Because then TRIM/discard isn't allowed because the holes would make it easiert to break the encryption because the attacker will know wht is data and what free space. And without TRIM/discard SMR HDDs and SSDs can't use the free space to optimize the performance so the disks will run as slow as a disk will do when completely filled (so no free space that could be used as CMR or SLC cache and no free space to move data around or be defragmented so it could be read/written faster later).

Thank you again.

It is HW RAID (OWC Mercury Elite Pro dual mini with two Mercury Extreme Pro 6G SSDs)

So I will be using LUKS (after studying it...). Still trying to grok everything in lvm/lvs/pvs man pages for now. Or I turn the HW RAID off and use it as separate disks and do ZFS.

I am now wondering about trim/fstrim/discard. As far as I understand it, it is (for NVMe? for SATA?) best to periodically run fstrim from crontab and I am now wondering about trim yes/no discard yes/no etc. when wanting to encrypt an external HW RAID based on two SSDs with LUKS.

So, my guess is now:
- Use LUKS from PVE's Debian to encrypt the external HW RAID (have to study LUKS still)
- Setup a regular fstrim (how regular?) in PVE's Debian
- Let the client OS be ignorant of all the encryption/trimming/etc going on(?) or make sure it doesn't do trim/discard because PVE's Debian is already doing that

 

[Dunuin]

Thank you again.

It is HW RAID (OWC Mercury Elite Pro dual mini with two Mercury Extreme Pro 6G SSDs)

So I will be using LUKS (after studying it...). Still trying to grok everything in lvm/lvs/pvs man pages for now. Or I turn the HW RAID off and use it as separate disks and do ZFS.

I am now wondering about trim/fstrim/discard. As far as I understand it, it is (for NVMe? for SATA?) best to periodically run fstrim from crontab and I am now wondering about trim yes/no discard yes/no etc. when wanting to encrypt an external HW RAID based on two SSDs with LUKS.

So, my guess is now:
- Use LUKS from PVE's Debian to encrypt the external HW RAID (have to study LUKS still)
- Setup a regular fstrim (how regular?) in PVE's Debian
- Let the client OS be ignorant of all the encryption/trimming/etc going on(?) or make sure it doesn't do trim/discard because PVE's Debian is already doing that

Jup, but keep in mind that by using TRIM/discard you will weaken your encryption. But without it your SSDs might get really slow. I for example got here a raid0 of 4x 500GB SATA SSDs and all I get is 30-40 MB/s because the SSDs don't really like it to run completely filled up. Had to only partition 80% of the disks capacity to get some usable performance which is still not great.

If I remember right LUKS will ask you if you want your zero data encrypted or not when creating your LUKS container.