Ceph OSD encryption

[paradox55]

I haven't messed with it yet, but I'm assuming the admin.key unlocks the encrypted OSD. In which case, that key is on every node? Or the monitor nodes? Anyway, point is - how would you go about securing your setup from unauthorized access in the event of a physical attack which iirc is the whole point of encrypting drives.

Thank you!

 

[Stoiko Ivanov]

Ceph OSD encryption is explained quite well in the ceph-docs: https://docs.ceph.com/docs/nautilus/ceph-volume/lvm/encryption/

when creating an ecrypted OSD the OSD submits 2 keys to the monitor:
* the actual LUKS-key used for decrypting/opening the device (this stays only with the monitor)
* one key to authenticate to the monitor (which in turn then sends the decryption key so that the OSD can be opened).

The actual LUKS key is not written onto the OSD itself.

how would you go about securing your setup from unauthorized access in the event of a physical attack which iirc is the whole point of encrypting drives.

If some attacker gains access to your running systems with enough privilege (physical access can lead to a lot of privilege - see e.g. https://en.wikipedia.org/wiki/Cold_boot_attack) there is nothing from stopping them to gain access to your data, since your data is live and accessible in the system (otherwise your guests would not be able to access their disk-drives).

OSD-encryption helps if the system is shut off, or a disk gets pulled out of the system.
It is also a great way to ensure that noone can salvage sensitive data from a broken disk, which you return for RMA (or recycle)

I hope that helps!

 

[paradox55]

Ceph OSD encryption is explained quite well in the ceph-docs: https://docs.ceph.com/docs/nautilus/ceph-volume/lvm/encryption/

when creating an ecrypted OSD the OSD submits 2 keys to the monitor:
* the actual LUKS-key used for decrypting/opening the device (this stays only with the monitor)
* one key to authenticate to the monitor (which in turn then sends the decryption key so that the OSD can be opened).

The actual LUKS key is not written onto the OSD itself.


If some attacker gains access to your running systems with enough privilege (physical access can lead to a lot of privilege - see e.g. https://en.wikipedia.org/wiki/Cold_boot_attack) there is nothing from stopping them to gain access to your data, since your data is live and accessible in the system (otherwise your guests would not be able to access their disk-drives).

OSD-encryption helps if the system is shut off, or a disk gets pulled out of the system.
It is also a great way to ensure that noone can salvage sensitive data from a broken disk, which you return for RMA (or recycle)

I hope that helps!

Right, I'm more concerned about physical attacks and like you mentioned sensitive data.

So each monitor would need to have root file system encryption to secure the keys, correct?

 

[Stoiko Ivanov]

So each monitor would need to have root file system encryption to secure the keys, correct?

This protects the data from the case where an attacker gets access to the disk(s) the monitor stores its data on, and to encrypted OSD-disks.

It won't help if an attacker gets privileged access to your running system (the encryption keys need to be in memory after all)

But I would probably keep the whole system encrypted for the gained benefit

I hope this helps!

 

[RedBlood9]

Hi to All,
I know this thread is not being updated for a while, but I think my question is related
We are installing a new ceph cluster on our proxmox 6 servers; and we would like to know where the key is saved in ceph, and we necessary to make some sort of backup or other particular attention to avoid data loss or anything else working with an encrypted cluster.

We would like to avoid to encounter problems of this kind once the server are in production.
Thanks

 

[Alwin]

We are installing a new ceph cluster on our proxmox 6 servers; and we would like to know where the key is saved in ceph, and we necessary to make some sort of backup or other particular attention to avoid data loss or anything else working with an encrypted cluster.

The key is stored in the MON DB and a cluster needs at least 3x MONs.

 

[RedBlood9]

Good to know, we are much calmer now.
Thanks for you quick reply!

 

[eduncan911]

Sorry to me a necromancer... But I have some additional questions on exactly this topic.

When securing a system, you evaluate the threat models first. So for the school I am setting this up at, physical access while a system is running is not of a concern.

What is a concern is theft of the physical servers by non-technical people (break-in, steal everything, hock it). Then the risk is only from cold boot of a 3rd party.

So I would classify our risk to be "from Cold Boot."

How would one mitigate this attack, preventing the encryption key from being retrieved from the OSD Monitors' DB? For example, if they were to just plug everything in and boot it.

Would encrypting just the OS boot drives be enough?

What about if one moves the Ceph OSD WAL/DB partitions on dedicated Enterprise SSD?

Is this what the RocksDB is used for? That seems like an easy restore-from-SSD block.db partition to a RocksDB format, and read - in clear-text?

---

Another way to look at it...

My company requires a Security Review Request for our InfoSec teams. They would absolutely require ownership of those encryption keys, most likely derived from a root key (if LUKS supports such a thing, AWS KMS does).

In the event they do not own the root certificate nor root key, clear documentation of precisely where the key(s) are stored is required which would require, you guessed it, it to be stored on an encrypted (data-at-rest) volume. In addition, they would require rotation of the keys on a 30day/1year cycle.

This is what a lot of organizations call Bronze-Level secure infrastructure, which is the lowest level requirements. Security audits also require precisely this information.

U.S. Government requires Gold level for our partnerships.

The point I am trying to make is, we need a lot more clearly spelled out, where and what format it is stored, how to retrieve and/or replace the keys, etc. I tried poking around in code, but wasn't able to quickly find a reference.

Since it's LUKS, rotation would be a simple matter of generating a new key and removing the old one.

However, RocksDB schema and how it is stored would be required.

It basically comes down to this: a RocksDB stored on an unencrypted partition would be meaningless if it contains the LUKS private keys to unlock the Ceph volumes/pools. The RocksDB would also need to be stored in an encrypted format with data-at-rest.

 

[ideamaster]

What is a concern is theft of the physical servers by non-technical people (break-in, steal everything, hock it). Then the risk is only from cold boot of a 3rd party.

So I would classify our risk to be "from Cold Boot."

https://www.trustedsys.com/images/stories/pdfs/TrustedSystems-CompanyBrochure.pdf
https://hamiltonproductsgroup.com/products/ips/

You basically keep your "SAN/NAS/DAS Head unit" in an IPS (10 minute protection against forced entry) and if an alarm happens it sends out an alert then kills power/SCRAM to x devices, maybe turns on a "space heater" to help the memory dissipate faster.

Get hardware with bios that supports "check memory on boot" in your bios
I would think you could even build something into the kernel as part a shutdown process to zero out physical memory if you were super paranoid..