Proxmox Ceph - Connect external workloads to proxmox Ceph

[breakaway9000]

Hi,

I’ve got a 3 node proxmox cluster running ceph.

I’m also running Kubernetes on top of proxmox.

Currently, my LAN is 172.16.254.0/24 and my Ceph network is 10.10.10.0/24 (separated over a different NIC and VLAN)

How do I give my Kubernetes access to Ceph? I want to create a separate pool and use object storage.

I guess I have to start with adding a virtual NIC for each Kube node into the 10.10.10.0/24 network so ther can communicate with the Proxmox Ceph Mons - am I on the right track here or is there a better way to do this?

 

[breakaway9000]

Turns out that is not the right way to do it. The right way is to update the config so that the OSDs are running on 10.10.10.0/24 network (dedicated 10gbps) and put the monitors/managers on your 172.16.254.0/24 (or whatever) network.

In the below scenario, I am setting up a "Public" network for my ceph in the 172.16.254.0/24 subnet (with the monitor IPs changing from 10.10.10.1-3 to 172.16.254.50-52).

Below is a quick-n-nasty way to do this. Note that this requires cluster downtime.

First, export the ceph monitor map to a file ("MONMAP_FILE" in this case).

ceph mon getmap -o MONMAP_FILE

Now, check what's inside it.

monmaptool --print MONMAP_FILE

Now, delete the old hosts & add the new hosts with the correct IP.

monmaptool --rm proxmox-node01 --rm proxmox-node02 --rm proxmox-node03
monmaptool --add proxmox-node01 172.16.254.50:6789 --add proxmox-node02 172.16.254.51:6789 --add proxmox-node03 172.16.254.52:6789 MONMAP_FILE

Check everything - make sure you got your IPs & hostnames correct.

Then, move the file to /etc/pve - corosync will now put it on all your hosts. You don't have to do it this way but I found it to be the easiest way to shift stuff between nodes quickly.

mv MONMAP_FILE /etc/pve

Now connect to your proxmox hosts, stop all VMs & CTs. Then, navigate to "ceph" and then to "Monitors", then stop all monitors. Now its time to inject the new monmap file into each host.

Run this command on each host - modify as needed of course

ceph-mon -i <hostname> --inject-monmap /path/to/MONMAP_FILE

For eg, I ran the following 3 commands

ceph-mon -i proxmox-node01 --inject-monmap /etc/pve/MONMAP_FILE
ceph-mon -i proxmox-node02 --inject-monmap /etc/pve/MONMAP_FILE
ceph-mon -i proxmox-node03 --inject-monmap /etc/pve/MONMAP_FILE

Once that's done, you need to edit /etc/pve/ceph.conf to include your new IP addressing. I have posted my FULL ceph.conf for reference.

[global]
         auth client required = none
         auth cluster required = none
         auth service required = none
         fsid = sdf-287b-46ff-a302-sdfasdf
         keyring = /etc/pve/priv/$cluster.$name.keyring
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 172.16.254.0/24
         cluster network = 10.10.10.0/24

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.proxmox-node01]
         host = proxmox-node01
         mon addr = 172.16.254.50:6789

[mon.proxmox-node02]
         host = proxmox-node02
         mon addr = 172.16.254.51:6789

[mon.aproxmox-node03]
         host = proxmox-node03
         mon addr = 172.16.254.52:6789

At this point, you're supposed to be able to restart the monitors and then the OSDs one by one and everything should just work (TM) but I couldn't get this working. I rebooted all 3 of my hosts - everything now functioning as normal.

I'd appreciate if anyone can chime in on why it doesn't work without a reboot.

References:
[1] http://docs.ceph.com/docs/mimic/rados/configuration/network-config-ref/
[2] http://docs.ceph.com/docs/mimic/rados/operations/operating/
[3] http://mohankri.weebly.com/my-interest/ceph-monitor-ip-address-changed
[4] https://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/
[5] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/002702.html
[6] http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/