Proxmox is sending backup to pbs in clear text when encryption file is unavailable

P

proxmox_is_awesome

New Member
Aug 13, 2022
8
0
1
Hello everyone, I just discovered something really odd regarding the encrypted backups to pbs and I wanted to share.

I have setup my encrypted backup to a locally hosted pbs instance.

The encryption key is stored in client at
etc/pve/priv/storage/<STORAGE-ID>.enc.

If you remove this file, backups still complete without error. I'm not sure at this point if the backups are encrypted. If backups are being sent as clear text just because the encryption file is missing for some reason, this seems like a terrible thing. As the user would have no idea that their data is now exposed.
 
Hi!

we prefer (potentially) security relevant issues to be reported privately so that we can take a look without rushing the rollout of fixes (we do try to acknowledge them and give initial feedback fast!). thanks for your report nevertheless - and sorry for the delayed response (yesterday was a federal public holiday here in Austria).

in this case, the scope of the issue (and the fix) is quite small:
- it only affects non-standard setups such as yours with the file being a symlink or being manually created/deleted - the encryption key files are supposed to be managed using pvesm/the corresponding API calls, which will always update both the file and the encryption-key config option for that storage
- a patch cross-checking the result of opening the encryption key with whether a fingerprint is set in the storage.cfg will be rolled out soon (after internal testing)

I'll update the thread once packages with the fix hit the repos.
 
  • Like
Reactions: proxmox_is_awesome, Dunuin and Neobin
Sorry about that initially I didn't think it would turn out to be a security issue. But later I realized that it was and tried to reach out as many places as I could.


Yes the scope of the issue is limited than general use case but it can happen to anyone. Honestly the way I see it a user is essentially one bad /mv/cp/rm command away from exposing their data. File can be removed for a thousand other reasons. A bad package update for example. And implications for this can be absolutely disastrous.


Thank you for taking the issue in. To conclude is there a way to achieve what I want safely (a custom secure location for the keys). Or should I open a new thread for that discussion?
 
proxmox_is_awesome said:
Sorry about that initially I didn't think it would turn out to be a security issue. But later I realized that it was and tried to reach out as many places as I could.
Click to expand...
can happen - forum posts can get lost in the noise, especially over weekends/holidays. a mail to office@proxmox.com with security report or something similar in the subject will very likely also get our attention outside of regular office hours ;) we're finalizing some stalled work on publishing concise information about how to report potential security issues that should help for similar occurrences in the future!
proxmox_is_awesome said:
Yes the scope of the issue is limited than general use case but it can happen to anyone. Honestly the way I see it a user is essentially one bad /mv/cp/rm command away from exposing their data. File can be removed for a thousand other reasons. A bad package update for example. And implications for this can be absolutely disastrous.
Click to expand...
yes, it could be triggered by a stray rm/mv by root - hence why we will roll out the fix fast. package update - not very likely (files in /etc/pve are not modified by packages directly, and any such change in a maintainer script would receive much scrutiny).
proxmox_is_awesome said:
Thank you for taking the issue in. To conclude is there a way to achieve what I want safely (a custom secure location for the keys). Or should I open a new thread for that discussion?
Click to expand...
we can continue discussion this here - your approach with a symlink doesn't work because pmxcfs (the fuse file system behind /etc/pve) doesn't support symlinks (other than a few baked-in ones where source and target are on pmxcfs). there's a reason for that - we want anything accessing files on /etc/pve to be reasonably sure that the contents are identical on all nodes, which would obviously not be the case if an admin could replace arbitrary paths with symlinks to local files/dirs. so the only way out would be to make the location configurable - but for a cluster-wide storage location the problem would still remain - what if the key path contains key A on one node, but key B on another?

the better solution would likely be to encrypt the underlying storage of /etc/pve (which is an sqlite DB in /var/lib/pve-cluster) - with the nice side-effect of taking care of all the possibly sensitive files stored there without the need for adding overrides for each and every one of them.
 
  • Like
Reactions: Dunuin
fabian said:
can happen - forum posts can get lost in the noise, especially over weekends/holidays. a mail to office@proxmox.com with security report or something similar in the subject will very likely also get our attention outside of regular office hours ;) we're finalizing some stalled work on publishing concise information about how to report potential security issues that should help for similar occurrences in the future!
Click to expand...
Sounds great. I actually sent an email there as well I believe.

fabian said:
the better solution would likely be to encrypt the underlying storage of /etc/pve (which is an sqlite DB in /var/lib/pve-cluster) - with the nice side-effect of taking care of all the possibly sensitive files stored there without the need for adding overrides for each and every one of them.
Click to expand...

Makes sense. How would I go about it though. Mounting a decrypted luks at /var/lib/pve-cluster is one option. But I am guessing this will cause services on start up to fail if dir is not there. Or even worse misbehave with old data. What would be your recommendation for this?

Ideally I would like PVE to be functional and running (I am already storing my vm disks on a thin lvm using this guide. Works well because PVE is up and running , I just need to unlock it to start the vms.) in this case as well (so I can remote it when not at home).
 
proxmox_is_awesome said:
Makes sense. How would I go about it though. Mounting a decrypted luks at /var/lib/pve-cluster is one option. But I am guessing this will cause services on start up to fail if dir is not there. Or even worse misbehave with old data. What would be your recommendation for this?

Ideally I would like PVE to be functional and running (I am already storing my vm disks on a thin lvm using this guide. Works well because PVE is up and running , I just need to unlock it to start the vms.) in this case as well (so I can remote it when not at home).
Click to expand...

yes, the usual approach would be an encrypted LUKS volume
- for '/', and using SSH-in-initramfs for remote unlocking (no other modifications needed)
- for '/var/lib/pve-cluster', requires a drop in file extending the pve-cluster service to make it wait on unlocking and mounting of the encrypted volume. everything else that's relevant in PVE's stack waits for pve-cluster anyway.

the second option has a caveat - root's authorized key file is a symlink to /etc/pve/priv/authorized_keys, so there's a kind of chicken and egg problem there. either use a non-root account (and su/sudo), or add a second authorized keys file (either via sshd config, or by creating /root/.ssh/authorized_keys2)
 
  • Like
Reactions: proxmox_is_awesome
Awesome. Do you happen to have any links to guides/examples for this? If not could you describe how to implement the 2nd option (just Googled the first option and what I found seemed to complicated and fragile) ?

Sorry to dump so much work on you but I'm not as proficient as I would like to be in these matters. So I appreciate your help a lot.


Also is there a future plan for in built fde. As in how Ubuntu allows for one click fde during os installation? Might be worth it to implement it like Android does it. It would load the signed os image on boot and happily run basic services (i.e allow you to receive calls etc) but until user enters passcode, user fs isn't mounted. A similar approach can be adopted here as well. Run basic admin services until fde is unlocked.
 
FDE as implemented using LUKS basically requires unlocking in initramfs (before any actual system services are started). anything other than that quickly gets very messy, especially if you allow the flexibility of extending the base system like PVE does.

w.r.t. encrypting /var/lib/pve-cluster - basically adding RequiresMountsFor=/var/lib/pve-cluster and adding a luks-encrypted partition and .mount unit for that path should work. obviously best to test it in a PVE-VM first ;)

you basically want the following "chain"
- pve-cluster.service -> requires a mount of /var/lib/pve-cluster (extension snippet via systemctl edit pve-cluster.service)
- /var/lib/pve-cluster mount unit -> requires cryptsetup unit to be active (can be generated via /etc/fstab, or written from scratch)
- cryptsetup unit -> requires underlying block device to be there, prompts for password (this can be generated via /etc/crypttab - see man systemd-cryptsetup-generator)
 
  • Like
Reactions: proxmox_is_awesome
  • Like
Reactions: Neobin, Stoiko Ivanov, proxmox_is_awesome and 1 other person
fabian said:
FDE as implemented using LUKS basically requires unlocking in initramfs (before any actual system services are started). anything other than that quickly gets very messy, especially if you allow the flexibility of extending the base system like PVE does.

w.r.t. encrypting /var/lib/pve-cluster - basically adding RequiresMountsFor=/var/lib/pve-cluster and adding a luks-encrypted partition and .mount unit for that path should work. obviously best to test it in a PVE-VM first ;)

you basically want the following "chain"
- pve-cluster.service -> requires a mount of /var/lib/pve-cluster (extension snippet via systemctl edit pve-cluster.service)
- /var/lib/pve-cluster mount unit -> requires cryptsetup unit to be active (can be generated via /etc/fstab, or written from scratch)
- cryptsetup unit -> requires underlying block device to be there, prompts for password (this can be generated via /etc/crypttab - see man systemd-cryptsetup-generator)
Click to expand...
I'll get this done and get back to you. I'll post a step by step guide here as well for others. Thank you so much.
 
available in pvetest now ;)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom