Proxmox and ceph encryption

[mxscbv]

Hello

I'm deploying a Proxmox/ceph cluster consisting of 3 physical nodes.

I'd like to encrypt the whole ceph storage. I know there is an 'Encrypt' checkbox when creating OSDs, but it encrypts/decrypts automatically as needed and only protects when a system is off/disk taken out of the node. But it doesn't protect if the whole server is taken out and then booted somewhere else.

I'd like to encrypt the storage with a passphrase (the same way I encrypt zfs datasets) so I can enter the passphrase manually on every boot to further protect data.

Is that possible? If yes, how do I achieve that?

Thanks,

Max

 

[SINOS]

I *assume* the OSD encryption keys are stored on the root filesystem and not on the OSD disk directly.
So, if you're using ZFS for your Proxmox Boot Disks, you could encrypt your rpool with a passphrase, which would effectively protect the OSD encryption keys in case the server is stolen.

 

[mxscbv]

Could you please advise how do I store the keys separately from the OSD disk?

 

[dcsapak]

the encryption secret for the osds are stored with the monitor (see https://docs.ceph.com/en/latest/ceph-volume/lvm/encryption/) so AFAIK only having the server with osds is not enough, you'd need also access to the monitors
it's not entirely clear from the docs how exactly the monitors store the key though..

 

[mxscbv]

I *assume* the OSD encryption keys are stored on the root filesystem and not on the OSD disk directly.

the encryption secret for the osds are stored with the monitor

Thanks for your input guys. Could you pls advise where these monitors/keys are physically stored with the monitors? Can I choose a specific dataset for them? The idea is to put these keys into an encrypted dataset.

 

[dcsapak]

the key should be stored in the monitor db and those are located at '/var/lib/ceph/mon'

 

[mxscbv]

the key should be stored in the monitor db and those are located at '/var/lib/ceph/mon'

So, if you're using ZFS for your Proxmox Boot Disks, you could encrypt your rpool with a passphrase

Thanks for your reply. How do I protect that key in an encrypted rpool? AFAIK I can't encrypt an existing pool/dataset, so the only option would be to create a new encrypted dataset and somehow put the keys there.

Could anyone pls advise what would be the best way to proceed with that?

Appreciate your help.

 

[mxscbv]

Anyone?

 

[mxscbv]

the key should be stored in the monitor db and those are located at '/var/lib/ceph/mon'

Is there a way to encrypt this folder somehow?

 

[Moonlander]

(Thanks for opening this thread - my intended use case is exactly yours!)

I think it should somehow be possible to create an additional encrypted ZFS dataset that uses the aforementioned path as a mount point and appropriately ingests the Ceph Monitor data. I'm just not sure if there is a way to delay the start of the Ceph Monitor service until the dataset is unlocked (either manually or somehow automatically, e.g. via another system on the network). I assume it's not a good idea to let the monitor start without being able to access the mon db - I guess some kind of error detection would go crazy then.

@dcsapak: Do you know of any options preventing that? Is mounting an encrypted ZFS dataset on top of the mentioned path a viable solution?

 

[mxscbv]

The folder /var/lib/ceph/mon is located in rpool. You can double check that by running:

df /var/lib/ceph/mon

You should receive something like:

Filesystem 1K-blocks Used Available Use% Mounted on
rpool/ROOT/pve-1 7082348288 8044928 7074303360 1% /

One of the possible solutions to this problem might be to encrypt the rpool completely (full disk encryption via LUKS). I believe we can achieve that by detaching the pool, running cryptsetup against it, and then attach it again. For additional convenience, I would also install dropbear to be able to remote SSH every time the Proxmox node restarts