Proxmox + PBS security issues

[glaeken2]

Scenario:
- One machine with proxmox
- Second machine with PBS

VMs on proxmox machine are DELETED, all the VM data is wiped.
After that, the proxmox machine is physically compromised.

What to expect: no data leak

---

Whan actually happens:
- PBS encryption key is available in /etc/pve on the compromised machine
- PBS user/token and password are also available there
- PBS IP address is also available.

Even if the PBS is properly secured, the threat actor can easily download backups from it, which results in practically non-existing security.

---

Current bugs which could mitigate this:
- PBS user Permission "Datastore.Backup" also allows for READS(!) of the backups, even if the "Datastore.Read" is NOT SET.

---

Solution for better backups. This is just a proposition:
Backup setup after proxmox install or adding pbs:
- Generate GPG public/private keypair. Let the user save the private key. On the proxmox itself keep only the public key.

Backup:
- On every VM backup: generate TEMPORARY aes key, encrypt it with the public key, and send it to the PBS
- Encrypt the VM backup with the temporary aes key, send it to the PBS.
- Delete the aes key from proxmox memory.

Restore:
- Ask the user to provide the private key
- Decrypt the temporary aes key for the given VM backup
- Decrypt and restore the vm.

 

[Hannes Laimer]

Hey,

if you delete VMs with the intend of destroying all data irreversibly then you necessarily have to also think about the backups you took, so you're responsible to destroy the encryption key in that case, or just delete the backups. Also, why would you assume the private key couldn't be compromised just because the user has it?

 

[glaeken2]

It's not only about destroying VMs, there are other cases:
What in case the VMs are not destroyed, but are on encrypted volumes and the key for those is volatile and only in memory? VMs will be safe, but backups not.

The problem is, the encryption key for PBS cannot be moved from /etc/pve (and database) to a safe place (i.e. external/network/encrypted/memory/manualentry place)

ps. The user can keep the private key safely encrypted. This is way harder to attack than just physical acquisition of the machine.

Question is really simple: how to mitigate access to backups after physical acquisition.
Reasons can be multiple, from encrypted VMs, deleted VMs, or older data which should not be accessible from backups.

[edit]
I've found another scenario which is even more simple:
- In case of hot-swap bays in rack servers, just pull the second drive from the raid out, read PBS access data from /var/lib database (for /etc/pve) and dump backups from PBS while the proxmox server is still running, so no one will even notice.
That's just a little security nightmare.

 

[Hannes Laimer]

Well, you don't. At least not if you want backup jobs that run without the user being present, for obvious reasons you can't use asymmetric encryption for the backup data itself, therefore the system needs to access the key without the user being present. If you feel like this is too unsafe, just empty the file containing the key after taking a backup and repopulated it again before you take a backup. Also keep in mind, if someone literally steals your server, they have direct access to the data, not just the backups.

You could also add Full Disk Encryption to PVE, there was a forum post[1] about that some time ago. I think that should solve most of your concerns, unless of course they have access to the root user and physically plug their keyboard into your server, assuming they can't over the network.

[1] https://forum.proxmox.com/threads/adding-full-disk-encryption-to-proxmox.137051/

 

[glaeken2]

If you feel like this is too unsafe, just empty the file containing the key after taking a backup and repopulated it again before you take a backup

This is only a partial solution, because when the backup is running, the key would be written to the database in /var/lib and so is present on the harddrive.
Also I don't know if the database for corosync does properly delete the key when the file is emptied.

And yes, you can use asymmetric encryption for backups, yes it's slow, but you can avoid slowness by encrypt the temporary aes key only, not the data itself which is then encrypted by the temporary key. This may create chunk deduplication problems on PBS, but it's a different topic.

In practical application you can actually encrypt logs on servers without user present, using asymmetric encryption, so the logs are only available when the user uses his private key. Where and how are different topics.

 

[spirit]

Maybe it could be great to be able to store the encrpytion key in a HSM device ?

like the new yubico hsm for example (not sure it's they are a standard lib for different devices)

https://www.yubico.com/la-cle-yubikey/yubihsm/?lang=en

 

[LnxBil]

You could also add Full Disk Encryption to PVE, there was a forum post[1] about that some time ago. I think that should solve most of your concerns, unless of course they have access to the root user and physically plug their keyboard into your server, assuming they can't over the network.

I thought of the same thing. This will solve all problems OP has.

 

[glaeken2]

Maybe it could be great to be able to store the encrpytion key in a HSM device ?

It would be enough to read it from for example /dev/shm (basically memory)
Or just keep it in memory: Process responsible for audit, checks if the proxmox filesystem is unaltered and if it is, it loads the key into memory. The end.
Keys for i.e. LUKS are not kept on the drive. This is almost the same situation because the PBS is "a network drive" basically.
Would anyone keep LUKS keys on the drive?

 

[glaeken2]

You could also add Full Disk Encryption to PVE, there was a forum post[1] about that some time ago. I think that should solve most of your concerns, unless of course they have access to the root user and physically plug their keyboard into your server, assuming they can't over the network.

Yes and no. Full disk encryption including root partition is a very hard thing to do with machines which have to be remotely reachable before they are decrypted. It's almost a super-experimental unstable setup.

I see some solutions here: moving the key from /etc/pve, hiding IP address of the PBS by iptables redirect or temporary domain?, changing backup type to asymmetric encryption, changing user permissions on PBS so the backups can't be read by the user (the user can only create backups, not read them).
Some are doable by the user, some would need an attention from the devs.

 

[LnxBil]

changing user permissions on PBS so the backups can't be read by the user (the user can only create backups, not read them).

You also need to check if a user is capable of deleting a backup. We had an incident which involved exactly this.

 

[glaeken2]

You also need to check if a user is capable of deleting a backup. We had an incident which involved exactly this.

Deletion is ok. If there is no permission for it, the user can't delete. Problem is only with reading. By default "Backup" permission includes "Read", even if "Datastore.Read" is a separate permission listed in the permissions list.

[edit]
Maybe a solution would be adding a permission "Datastore.NoRead" - this would not violate current behavior and it should be simple to add.

 

[fabian]

Datastore.Read is actually a higher privilege level, it allows reading the whole datastore.

Your proposed asymmetric encryption would basically nullify most of the PBS advantages - it breaks deduplication. At that point you can just implement your own encryption using regular vzdump to stdout and ditch PBS altogether

 

[Dunuin]

This is only a partial solution, because when the backup is running, the key would be written to the database in /var/lib and so is present on the harddrive.
Also I don't know if the database for corosync does properly delete the key when the file is emptied.

Here the whole root filesystem is encrypted including all the config.db files of all nodes with its pmxcfs. So the key is encrypted too. But of cause wont help when your host gets compromized while it is running and unlocked.

Yes and no. Full disk encryption including root partition is a very hard thing to do with machines which have to be remotely reachable before they are decrypted. It's almost a super-experimental unstable setup.

Here it works flawlessly for years. You can use ready-to-use solution to remote unlock the server like clevis or use something custom like unlocking the root filesystem via SSH and expect scripts.