Proxmox 5.2 + Ceph Luminous

breakaway9000

Renowned Member
Dec 20, 2015
91
21
73
Hi,

I'm just wondering if with proxmox, running ceph monitors in a different subnet is a supported configuration? The official guide that walks you through configuring monitors only gives the option to create OSDs running on the nodes themselves, i..e on the one subnet (as defined by the "pveceph init --network" command)

The reason I want to run the monitors in a different subnet is I want them at a different site due to encryption. I have read in the ceph documentation that you can encrypt each ceph OSD however the key management is provided by the monitor (i.e. the keys are stored on the monitor itself).

This seems kind of unsafe to me - what good is storing the key right there on the server that has the disk you want to secure in it?

So in a nutshell, is it a supported config to run the monitors in a different subnet from a different site with slightly higher than LAN latency (between 1-5ms).

If not, what is the best way to accomplish secure encryption?
 
This seems kind of unsafe to me - what good is storing the key right there on the server that has the disk you want to secure in it?

your mon nodes and your osd nodes don't need to be the same physical systems. but the main reason for encrypting OSD disks is that you can just throw them away when they are broken or get taken out of service, and don't need to worry about someone being able to recover the data. you just need to take care that you destroy the mon disks (which store the LUKS keys) securely. note that nothing prevents you from manually unlocking your OSD disks on every reboot instead of providing the LUKS passphrase via the monitors - most datacenter deployments don't want this level of manual interaction though.
 
Hi Fabian, Can we use "pveceph init --network" to create the ceph public and cluster network in the same time?
 
Or, just need to change after "pveceph init --network" with ceph public network CIDR, then change "/etc/ceph/ceph.conf" of one of the nodes, change the "cluster network" to a private network CIDR, then pmxcfs will sync this file all across the cluster?
 
Or, just need to change after "pveceph init --network" with ceph public network CIDR, then change "/etc/ceph/ceph.conf" of one of the nodes, change the "cluster network" to a private network CIDR, then pmxcfs will sync this file all across the cluster?

yes, this works.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!