Encryption Options for VM´S /Server

[Pampa Party]

Hi,

i try fo find a strategy to encrypt my vm´s or my server. I found some Topics here about but not really what i´m looking for. For me the best solution would be to encrypt every vm by itself, so the whole system, so the backups are also encrypted. I tried with veracrypt but failed. The encryption works but i cannot acces the vm afterwards. I guess theres a Problem with hoe Proxmox handle Storage or the boot order?! Anybody has an Idea what i can do?

Reasons for me to use veracrypt is, cause i know it and i want to be able te decrypt in case i need to add some more space to the vm. In my mind that would be the easiest workflow for my use cases so far.

I know there are possibilities just to add encrypted storage, but i would prefer to encrypt the whole vm. Anybody has a workaround here to mention?

I also seen LUKS to use. I tried this to but when i want to encrypt i get asked if i want to delete all my current data on the storage. But i could lacking of knowledge here.

So if anybody has a workaround for an easy way to encrypt vms in proxmox, or the whole proxmox server, t would be lovely to let me know .

Best regards

 

[esi_y]

I also seen LUKS to use. I tried this to but when i want to encrypt i get asked if i want to delete all my current data on the storage. But i could lacking of knowledge here.

I would just use LUKS on the host, what exactly is your issue with that?

 

[Dunuin]

For me the best solution would be to encrypt every vm by itself, so the whole system, so the backups are also encrypted.

For encrypted backups you could setup a PBS. Highly recommended anyway because it saves tons of space, is way faster and lots of nice features like live restore, syncing and ransomware protection.

 

[Pampa Party]

I would just use LUKS on the host, what exactly is your issue with that?

My Issue here is that, if i get i right so far, i need to enrypt the storage before i install for example a Server debian Operating system. I didnt´t find a solution so far to enrypt an existing VM without data loss. But as i said im not familiar with LUKS so far, so i could be wrong here. If you have a suggestion for me you´re welcome to share . Also is there for me the point when i want to resize the storage later. How will this works with luks

For encrypted backups you could setup a PBS. Highly recommended anyway because it saves tons of space, is way faster and lots of nice features like live restore, syncing and ransomware protection.

For Backups i guess i´ll try sooner or later. But i can´t get my vm´s encryptet with PBS.

 

[UdoB]

But i can´t get my vm´s encryptet with PBS.

For sure you can!

It is well integrated in the GUI: you'll find a page "Encryption" in the dialog of "Add: Proxmox Backup Server" dialog. These keys are client-side only --> the server is not trusted and may be hosted externally by an untrusted company...

For CLI see https://pbs.proxmox.com/docs/backup-client.html#encryption

 

[Pampa Party]

For sure you can!

It is well integrated in the GUI: you'll find a page "Encryption" in the dialog of "Add: Proxmox Backup Server" dialog. These keys are client-side only --> the server is not trusted and may be hosted externally by an untrusted company...

For CLI see https://pbs.proxmox.com/docs/backup-client.html#encryption

thx for the reply. I understood it like that this is the encryption just for the backups, not the vm itself?! But if i can also encrypt my vm´s i´ll give it a try

 

[esi_y]

My Issue here is that, if i get i right so far, i need to enrypt the storage before i install for example a Server debian Operating system.

Well, you need a partition and the space wiped before you make it into LUKS. Since PVE install is basically immutable (consider when you upgrade cluster nodes, you are basically keeping just your config), you might as well install that fresh (on a separate partition).

You may start assessing LUKS by looking at a thread like this:
https://forum.proxmox.com/threads/proxmox-8-luks-encryption-question.137150/#post-610704

I didnt´t find a solution so far to enrypt an existing VM without data loss.

You simply move them over onto encrypted partition or your pull them there from backups. Since it's valuable data that needs encryption, you do have (separate space for) backups, right?

But as i said im not familiar with LUKS so far, so i could be wrong here. If you have a suggestion for me you´re welcome to share .

I think you are looking for any encryption at rest, this would be transparently best done with LUKS or if your hardware allow,s self-encrypting drives (SED) sestup.

Also is there for me the point when i want to resize the storage later. How will this works with luks

I don't like to be answering someone's question by first telling them to do it differently, because I do not know why they ask. So with LUKS, it's just a layer, if you have LVM or ZFS above it, you can resize (provided you know what you do with that particular system). I suppose you are asking mostly about increasing size and that really is not a problem and does not depend on LUKS but on that ZFS or LVM above.

That said, I would never have to resize anything, I should be able to pull it from backups on a freshly repartitioned drive (array) at any point, because that's what I have to be able to do for any disaster recovery scenario anyways and it's less time consuming and less error prone.

For Backups i guess i´ll try sooner or later. But i can´t get my vm´s encryptet with PBS.

Answered separately by @UdoB - I would just add (having no experience with PBS myself, I do not like it ) - I would have LUKS on that storage on which I back up onto anyhow. But any backup system allows you to encrypt, even if only by piping it through some GPG. Think of e.g. duplicity backups that you than push onto public cloud where you do not trust their encryption at rest. Same principle.

 

[UdoB]

I understood it like that this is the encryption just for the backups, not the vm itself?!

It is for backup of Containers and VMs. Coming from a PVE host the backups are encrypted locally on that (EDIT: PBS ) PVE!- and only then it is sent to the PBS. The PBS only gets the encrypted data.

From my point of view this is a great concept

Especially because the other nice features like massive de-duplication (I see a factor of ~28) and live-restore are compatible to this, they work flawlessly.

The PBS host itself is not affected by the above. It may get installed on any base machine, including any below-the-os-encryption (LUKS etc.) The recommendation for having a local machine instead of a VM and having good SSDs instead of rotation rust stay valid or course

 

[esi_y]

The PBS only gets the encrypted data.

The PBS host itself is not affected by the above. It may get installed on any base machine, including any below-the-os-encryption (LUKS etc.)

This might not be a concern for the OP, but the PVE node must be affected by this if backing up (and deduplicating) large number of VMs and often, even if encryption is assisted by AES-NI, must it not?

The recommendation for having a local machine instead of a VM and

I would just say VM for anything backup is not a problem as long as it is not a VM on some of those nodes being backed up.

having good SSDs instead of rotation rust stay valid or course

I do not know about benefits of SSDs for backup specifically, i.e. if there's any benefit for cold storing any data - after all, it's just going to be sitting there and for the same purchasing price you get a mirror and more capacity in total (or cash to spare). Restore by definition must be sequential anyhow? Failure rate arguably for anything is 50% because if it fails it fails and if you can't afford it higher MTBF won't make you sleep better.

 

[UdoB]

This might not be a concern for the OP, but the PVE node must be affected by this if backing up (and deduplicating) large number of VMs and often, even if encryption is assisted by AES-NI, must it not?

Well, I was trying to say "the encryption of the PBS host and the datastore used for the actual backup data are two separate things."

You can encrypt the installation of the PBS (if you know how to) while the datastore gets unencrypted data. And vice versa ;-)

The presence and/or the requirement of AES-NI I have never tested nor have I checked its presence, sorry.

I would just say VM for anything backup is not a problem

Sure! In my understand it just is not recommended. Although... check..., https://pbs.proxmox.com/docs/installation.html#system-requirements does not mention virtualization at all. Actually I run two instances virtualized and three bare metal

I do not know about benefits of SSDs for backup specifically

You are right, this is not as critical as for the VM storage on PVE. I would recommend to use good SSDs because we are a) talking about a backup system - it must be reliable deliver my written data when an incident happens, b) it must write a lot of data (depending on changed data in the VMs on PBS - possibly every few hours, depending on the backup interval), c) requiring good write performance, d) it should have good IOPS for both writing and reading; reading is relevant for "Verify" and for "Restore" of course. Most SSDs qualify, while rotating rust is never recommended anymore.

But yeah, with "Verify Jobs" checking for consistency in a regular interval the storage in PBS may be simpler/cheaper than in PVE. While I have SSD-only storage for my primary PBS most of the others have a mixed setup = some ZFS mirrors on rotating plus plus a "Special Device", also mirrored on SSD.

(I would never omit the basic redundancy of those mirrors, even if some people may validly say "hey, it's just a backup = no redundant storage required". )

There are so many ways to skin a cat, but recommended is always the "will certainly not fail in under any normal circumstances" variant.

 

[Veidit]

Is it this your are looking for when it comes to encrypted VMs?
https://kvm-forum.qemu.org/2021/SEV-SNP-AMD-Brijesh-Singh.pdf

 

[esi_y]

Sure! In my understand it just is not recommended. Although... check..., https://pbs.proxmox.com/docs/installation.html#system-requirements does not mention virtualization at all. Actually I run two instances virtualized and three bare metal

On these "official documents", I just cannot help this, but "Backup storage: Prefer fast storage that delivers high IOPS for random IO workloads" ... what kind of random workloads with high IOPS does a backup server get to do when pushing backups? Then it goes on to say "metadata cache is highly recommended" which would point to what they are concerned about. I do not know the intricacies of how PBS works (other than it completely avoids using ZFS built-in snapshots for doing deltas at which point I gave up with it), but if the metadata needs high IOPS random access they better be stored on another partition (or are they)? Are these metadata all that much random accessed over the network since PVE node is doing all of the work?

Sorry if it appears I am going off the track here, but in the light of OP's concern (probably low on space / budget constraints) and also what one expects from a well designed backup system (long-term storage, imagine tape device even), the PBS would then not be very mature if it cannot make use of low IOPS medium for sequentially storing a stream of data on basically any medium.

If I look at "Memory (RAM): 2 GB RAM" requirement, I would argue it would be better off on 16GB RAM and a RAM disk (instead of ZFS special device) with as many spinning drives as one wants.

You are right, this is not as critical as for the VM storage on PVE. I would recommend to use good SSDs because we are a) talking about a backup system - it must be reliable deliver my written data when an incident happens, b) it must write a lot of data (depending on changed data in the VMs on PBS - possibly every few hours, depending on the backup interval), c) requiring good write performance, d) it should have good IOPS for both writing and reading; reading is relevant for "Verify" and for "Restore" of course. Most SSDs qualify, while rotating rust is never recommended anymore.

The thing with spinning drives is that they have "unlimited" TBW by definition, something great for lots of sequential writes of large amounts of data.

But yeah, with "Verify Jobs" checking for consistency in a regular interval the storage in PBS may be simpler/cheaper than in PVE. While I have SSD-only storage for my primary PBS most of the others have a mixed setup = some ZFS mirrors on rotating plus plus a "Special Device", also mirrored on SSD.

Don't get me wrong, there's nothing wrong with your setup, but it's not as cost effective as it could be (that all without sacrificing reliability).

(I would never omit the basic redundancy of those mirrors, even if some people may validly say "hey, it's just a backup = no redundant storage required". )

That's the thing, one still has to have redundancy, which with SSDs adds the cost factor.

There are so many ways to skin a cat, but recommended is always the "will certainly not fail in under any normal circumstances" variant.

Just sometimes I feel like these requirements are just a way to avoid having to create a well designed solution, it's the cheapest answer a developer can give - add more X/Y/Z.

 

[UdoB]

That's the thing, one still has to have redundancy, which with SSDs adds the cost factor.

Well, redundancy on vdev-Level is not required at all.

It is my personal choice: there is the technical opportunity to get redundancy and I really want to have this. I am fine to pay the double price for an easy recovery expectation.

Again: there is a wide, wide area between "recommended" and "works also".

Some people run PVE on a Raspberry Pi. This is not recommended and not officially supported at all. But if it works... it works. In a company you can not do this. In a Homelab it's completely up to you.

Have fun!

 

[esi_y]

Well, redundancy on vdev-Level is not required at all.

What do you mean? If you have a single vdev pool, what kind of backup is that? I mean, aside cases where you have 2nd, 3rd, etc backup elsewhere at the same time.

 

[Veidit]

What do you mean? If you have a single vdev pool, what kind of backup is that? I mean, aside cases where you have 2nd, 3rd, etc backup elsewhere at the same time.

There is a difference between what is required and what works, just as UdoB posted earlier.

 

[esi_y]

There is a difference between what is required and what works, just as UdoB posted earlier.

Ok I get that part, but having a backup which is on a single medium (with all the snapshots you may want to revert to) ... and then at some point discovering the backup is not usable (when you need it, even as it might have happened to it between the last verification and now) ... that's not a backup at all. I would even argue it's worse than having no backup in such case and be fully aware of it (so that anyone administering the VMs knows to e.g. back up valuable data by other means).

 

[Veidit]

Ok I get that part, but having a backup which is on a single medium (with all the snapshots you may want to revert to) ... and then at some point discovering the backup is not usable (when you need it, even as it might have happened to it between the last verification and now) ... that's not a backup at all. I would even argue it's worse than having no backup in such case and be fully aware of it (so that anyone administering the VMs knows to e.g. back up valuable data by other means).

But that is the responsebility of the administrator to know.
Proxmox can't hold your hand with everything, it's the administrators job to read the documentation and do a good installation.

 

[Pampa Party]

Thx for all teh replys and help so far . I guess i got a solution for encryption which will work so far with Luks.

Is there a way to automate the mnt process of the Luks container? Anybody has a workaround with tpm for example? After i reboot my system i need to encrypt the luks container, and mnt it to the mnt directory. i tried with fstab but as expected it failed because it wasn´t unencrypted when it comes to the mount process.

Any help is welcome as usual

 

[UdoB]

Is there a way to automate the mnt process of the Luks container?

You may possibly want to search for TANG and CLEVIS.

This toolset can unlock the LUKS key via your local and trusted network. If the server is stolen, it will probably not be able to establish a contact to your TANG. So CLEVIS cant NOT unlock the keys - the data stays encrypted.

 

[esi_y]

Is there a way to automate the mnt process of the Luks container? Anybody has a workaround with tpm for example? After i reboot my system i need to encrypt the luks container, and mnt it to the mnt directory. i tried with fstab but as expected it failed because it wasn´t unencrypted when it comes to the mount process.

Any help is welcome as usual

Not sure I understood what you are asking, but but with LUKS block devices you usually make use of /etc/crypttab [1]:

Or you might be looking for solution like dropbear [2]?

See e.g.: https://neilzone.co.uk/2023/05/unlocking-a-luks-encrypted-partition-via-ssh-on-debian-12-bookworm/

[1] https://www.freedesktop.org/software/systemd/man/latest/crypttab.html
[2] https://matt.ucc.asn.au/dropbear/dropbear.html