Encryption key storage

[carsten2]

1) Where does PBS store the encryption key in a cluster?
I have a proxmox cluster with 3 nodes and created a backup key on node1 as well as a master key. The documentation says, the keys are in /root/.config/proxmox-backup.
The corresponding directory in node2 has only the file "fingerprint" and still produces encrypted backups according to the GUI. So where does the proxmox-client on node2 get the encryption key from and where are they stored?

2) I use full data disk encryption to guard against server theft, but no system encryption because this is not well supported by proxmox. I want to make sure that the backup encryption key is not stored at the unencrypted system volume. I created a symlink from ./config/proxmox-backup to an encrypted datadisk to ensure the key is not readable in case of server theft. Is is safe or does proxmox-backup stores the key anywhere in the unencrypted system (oder corosync) file system?

 

[t.lamprecht]

Hi!

I have a proxmox cluster with 3 nodes and created a backup key on node1 as well as a master key. The documentation says, the keys are in /root/.config/proxmox-backup.

If you use the client directly and tell it to create a key without an explicit path, then yes, that is the location used.

But on PVE multiple nodes need to access that key, so it's stored in the clustered configuration filesystem /etc/pve, to be more specific in /etc/pve/priv/storage/<STORAGE-ID>.enc as it's documented in the Proxmox VE storage documentation - which is the relevant one if you use the PVE integration:
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_configuration_4

Note also, the integration is still a bit lacking in terms of key identifying and UX for key safe-keeping; that will be made stable once Proxmox VE 6.3 gets released in the upcoming weeks.

2) I use full data disk encryption to guard against server theft, but no system encryption because this is not well supported by proxmox. I want to make sure that the backup encryption key is not stored at the unencrypted system volume. I created a symlink from ./config/proxmox-backup to an encrypted datadisk to ensure the key is not readable in case of server theft. Is is safe or does proxmox-backup stores the key anywhere in the unencrypted system (oder corosync) file system?

You cannot make symlinks arbitrary in the pmxcfs (/etc/pve).

The encryption actually safes you from a not fully trusted Backup Server, or a break-in/theft of that backup server.
If one steals your PVE server physically or compromises it, they already have live access to all the VMs and CTs and their data, no point in taking the key then if one can just read all current data anyway. That's why you also should create different keys per cluster / stand-alone nodes, as then a compromise of any of those clusters does not put the others at risk.

If you're concerned about physical stealth and/or your PVE being compromised the actual way to protect both, the guests live data and the encryption key would be to encrypt the guest storage and the whole PVE root storage, or at least relevant partitions.
That then should include the path where the pmxcfs backing database and backups of that are stored: /var/lib/pve-cluster

 

[carsten2]

If one steals your PVE server physically or compromises it, they already have live access to all the VMs and CTs and their data, no point in taking the key then if one can just read all current data anyway. That's why you also should create different keys per cluster / stand-alone nodes, as then a compromise of any of those clusters does not put the others at risk.

The data disk are encrypted (my manually typing in a password), so stealing the servers (and switching it off of cause) keeps the data safe. The problem however is data or passwords which resides on the unencrytped system volume. To keep data save the passwords and keys should be on encrypted volumns also. Even better would be a full disk encryption of the system itsself. Unfortunately rpool encryption is not that easy with proxmox. It would be really nice if this would be supported (and/or documented).

Is there a way to encrypt the cluster fs? E.g. by linking /var/lib/lib-cluster to an encrypted file system?

 

[t.lamprecht]

Is there a way to encrypt the cluster fs? E.g. by linking /var/lib/lib-cluster to an encrypted file system?

Yes that'd work, but you need to get it unencrypted at boot before the pve-cluster.service starts up., as else that start will fail and with it multiple dependent important services.

 

[carsten2]

Yes that'd work, but you need to get it unencrypted at boot before the pve-cluster.service starts up., as else that start will fail and with it multiple dependent important services.

At boot decryption is not possible (otherwise I could use full rpool encryption). So is it possible to prevent pve-cluster.service to automatically start up and wait for manual disk decryption and manuall start command (all via ssh)?

 

[Ovidiu]

I just wanted to say that the wiki is slightly incorrect regarding the storage location of the encryption key.

https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server says:

encryption-key
A key to encrypt the backup data from the client side. Currently only non-password protected (no key derive function (kdf)) are supported. Will be saved in a file under /etc/pve/priv/<STORAGE-ID>.enc with access

It should be

 /etc/pve/priv/storage/<STORAGE-ID>.enc

 

[t.lamprecht]

At boot decryption is not possible (otherwise I could use full rpool encryption).

May not be straight forward, but it is possible.
https://forum.proxmox.com/threads/native-encryption-of-zfs-root-possible.56413/#post-259870

So is it possible to prevent pve-cluster.service to automatically start up and wait for manual disk decryption and manuall start command (all via ssh)?

You can add a systemd dependency which orders before pve-cluster.service, has infinite start timeout and only is finished (started up) once you entered the PW.

 

[t.lamprecht]

I just wanted to say that the wiki is slightly incorrect regarding the storage location of the encryption key.

https://pve.proxmox.com/wiki/Storage:_Proxmox_Backup_Server says:

encryption-key
A key to encrypt the backup data from the client side. Currently only non-password protected (no key derive function (kdf)) are supported. Will be saved in a file under /etc/pve/priv/<STORAGE-ID>.enc with access

It should be

 /etc/pve/priv/storage/<STORAGE-ID>.enc

True, it initially wasn't in a storage namespaced folder, password file location was also outdated. Fixed both, thanks!

 

[carsten2]

May not be straight forward, but it is possible.
https://forum.proxmox.com/threads/native-encryption-of-zfs-root-possible.56413/#post-259870

You can add a systemd dependency which orders before pve-cluster.service, has infinite start timeout and only is finished (started up) once you entered the PW.

The problem is not LUKS vs ZFS encryption, but the problem to be able to remotely unlock the server. The servers usually have no keyboard/monitor. So a base system is needed to boot up, connect via SSH to unlock the volumes.