Ceph (luks) password?

[proxwolfe]

Hi,

I set up a Ceph cluster and everything seems to be working fine.

There was the option to encrypt an OSD. I understand that, when chosen, this is done via LUKS. Which is great.

But now I am wondering how to access the contents of my disk (OSD) outside of Ceph, should I ever need to.

Is the password stored somewhere where I can find it? What else would be necessary to access the disk in Linux?

Thanks!

 

[proxwolfe]

Any ideas?

 

[jsterr]

did you check out https://docs.ceph.com/en/latest/ceph-volume/lvm/encryption/

When setting up the OSD, a secret key will be created, that will be passed along to the monitor in JSON format as stdin to prevent the key from being captured in the logs.

You should need:

"cephx_secret": CEPHX_SECRET,
"dmcrypt_key": DMCRYPT_KEY,
"cephx_lockbox_secret": LOCKBOX_SECRET,

 

[proxwolfe]

Thanks!

So key should be somewhere on a monitor node, if I understand this correctly. But how do I extract it (assuming this is possible at all)?

 

[jsterr]

Thanks!

So key should be somewhere on a monitor node, if I understand this correctly. But how do I extract it (assuming this is possible at all)?

If you are interested how Proxmox setups a a encrypted osd see here:

create OSD on /dev/nvme2n1 (bluestore)
wipe disk/partition: /dev/nvme2n1
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.2679 s, 783 MB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 53133f72-741b-44b0-b26c-3de5efb3b787
Running command: /sbin/vgcreate --force --yes ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9 /dev/nvme2n1
 stdout: Physical volume "/dev/nvme2n1" successfully created.
 stdout: Volume group "ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9" successfully created
Running command: /sbin/lvcreate --yes -l 228928 -n osd-block-53133f72-741b-44b0-b26c-3de5efb3b787 ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9
 stdout: Logical volume "osd-block-53133f72-741b-44b0-b26c-3de5efb3b787" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9/osd-block-53133f72-741b-44b0-b26c-3de5efb3b787
Running command: /sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9/osd-block-53133f72-741b-44b0-b26c-3de5efb3b787 9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/mapper/9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb
Running command: /bin/chown -R ceph:ceph /dev/dm-3
Running command: /bin/ln -s /dev/mapper/9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb /var/lib/ceph/osd/ceph-3/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap
 stderr: 2021-08-27T07:46:23.655+0200 7fdc6ae33700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2021-08-27T07:46:23.655+0200 7fdc6ae33700 -1 AuthRegistry(0x7fdc6405aed8) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: got monmap epoch 4
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring --name osd.3 --add-key AQAkfChh7HcGCRAAr7xp+g7JOSLW15sgXlQ7CA==
 stdout: creating /var/lib/ceph/osd/ceph-3/keyring
added entity osd.3 auth(key=AQAkfChh7HcGCRAAr7xp+g7JOSLW15sgXlQ7CA==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 53133f72-741b-44b0-b26c-3de5efb3b787 --setuser ceph --setgroup ceph
 stderr: 2021-08-27T07:46:23.895+0200 7f4edb6dbf00 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: /dev/nvme2n1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-3/lockbox.keyring --create-keyring --name client.osd-lockbox.53133f72-741b-44b0-b26c-3de5efb3b787 --add-key AQAkfChhkPG9CRAAQzPByZHTw+KHkAZApLrH3Q==
 stdout: creating /var/lib/ceph/osd/ceph-3/lockbox.keyring
added entity client.osd-lockbox.53133f72-741b-44b0-b26c-3de5efb3b787 auth(key=AQAkfChhkPG9CRAAQzPByZHTw+KHkAZApLrH3Q==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/lockbox.keyring
Running command: /bin/ceph --cluster ceph --name client.osd-lockbox.53133f72-741b-44b0-b26c-3de5efb3b787 --keyring /var/lib/ceph/osd/ceph-3/lockbox.keyring config-key get dm-crypt/osd/53133f72-741b-44b0-b26c-3de5efb3b787/luks
Running command: /sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-d05f6fbb-53e6-4344-bb10-bc0d67cad8d9/osd-block-53133f72-741b-44b0-b26c-3de5efb3b787 9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb
 stderr: Device 9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb already exists.
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb --path /var/lib/ceph/osd/ceph-3 --no-mon-config
Running command: /bin/ln -snf /dev/mapper/9PQi9h-3G48-cpGT-WgxN-ajXT-T7Go-2WlFyb /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -R ceph:ceph /dev/dm-3
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3
Running command: /bin/systemctl enable ceph-volume@lvm-3-53133f72-741b-44b0-b26c-3de5efb3b787
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-3-53133f72-741b-44b0-b26c-3de5efb3b787.service -> /lib/systemd/system/ceph-volume@.service.
Running command: /bin/systemctl enable --runtime ceph-osd@3
 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@3.service -> /lib/systemd/system/ceph-osd@.service.
Running command: /bin/systemctl start ceph-osd@3
--> ceph-volume lvm activate successful for osd ID: 3
--> ceph-volume lvm create successful for: /dev/nvme2n1
TASK OK

I tested it on my testing system and what I got (according to ceph mailing list ) is this:

root@pve03:~# /usr/bin/ceph config-key ls

[
    "config-history/1/",
    "config-history/2/",
    "config-history/2/+mon/auth_allow_insecure_global_id_reclaim",
    "config-history/3/",
    "config-history/3/+mgr/mgr/telegraf/address",
    "config-history/4/",
    "config-history/4/+mgr/mgr/telegraf/interval",
    "config/mgr/mgr/telegraf/address",
    "config/mgr/mgr/telegraf/interval",
    "config/mon/auth_allow_insecure_global_id_reclaim",
    "device/WUS4BB096D7P3E4_A069DF9B",
    "device/WUS4BB096D7P3E4_A069DFA0",
    "device/WUS4BB096D7P3E4_A069DFA1",
    "device/WUS4BB096D7P3E4_A069DFA2",
    "device/WUS4BB096D7P3E4_A069DFA5",
    "device/WUS4BB096D7P3E4_A069DFA6",
    "device/WUS4BB096D7P3E4_A069DFA7",
    "device/WUS4BB096D7P3E4_A069DFA9",
    "device/WUS4BB096D7P3E4_A069DFAA",
    "device/WUS4BB096D7P3E4_A069DFAC",
    "device/WUS4BB096D7P3E4_A069DFAE",
    "device/WUS4BB096D7P3E4_A069DFB2",
    "dm-crypt/osd/159ae3ff-6f00-4800-b665-b9e535519569/luks",
    "dm-crypt/osd/53133f72-741b-44b0-b26c-3de5efb3b787/luks",
    "mgr/crash/crash/2021-05-31T05:56:42.114601Z_47569ca5-f275-4ae9-afd7-3182c652724f",
    "mgr/dashboard/accessdb_v2",
    "mgr/dashboard/crt",
    "mgr/dashboard/jwt_secret",
    "mgr/dashboard/key",
    "mgr/devicehealth/last_scrape",
    "mgr/progress/completed",
    "mgr/telemetry/report_id",
    "mgr/telemetry/salt"
]

root@pve03:~# /usr/bin/ceph config-key get dm-crypt/osd/159ae3ff-6f00-4800-b665-b9e535519569/luks
KD0e+DGPOaAzBPkmo8o5qy/tcidbuiCwSMwkCKs6t3+zJlcbHEK2bquYAzZuqB06nKZCrCEoC9P6fo0lQDbPtfXzRR4uAoCsWjRJJSgfyJpnfxGhjWJOdd4qm+MzPqwPVQdmYHNWxC+WE5iB3PSwE0u7bMjR0uBmeQ2SfulgLEM=root@pve03:~#

root@pve03:~# /usr/bin/ceph config-key get dm-crypt/osd/53133f72-741b-44b0-b26c-3de5efb3b787/luks
PDvToPXnZ0Otfh4lxsjluaXdknl4/hyUX6ygMer9y+dB2nKjEqKVxsMD6TOAxFUmspihLDM99OuBQFe0gkaZtRKT4G7NR1+waT87uC1HWD6Td3zYGC6CbUEc65r5ZNp7doj4zLzaOmN26/ixibayRIwDO3V/xv9kBS0TH5s8XiE=root@pve03:~#

Sources:
https://forum.proxmox.com/threads/ceph-osd-verschlüsselung-mit-manueller-key-eingabe.72933/
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-April/026288.html
https://lists.ceph.io/hyperkitty/li...d/6GH2I2Q7HJWR5TYWCFADXD5Z2FQGWALF/?sort=date