Proxmox Ceph - Connect external workloads to proxmox Ceph

Discussion in 'Proxmox VE: Networking and Firewall' started by breakaway9000, Sep 13, 2018.

  1. breakaway9000

    breakaway9000 Member

    Joined:
    Dec 20, 2015
    Messages:
    49
    Likes Received:
    4
    Hi,

    I’ve got a 3 node proxmox cluster running ceph.

    I’m also running Kubernetes on top of proxmox.

    Currently, my LAN is 172.16.254.0/24 and my Ceph network is 10.10.10.0/24 (separated over a different NIC and VLAN)

    How do I give my Kubernetes access to Ceph? I want to create a separate pool and use object storage.

    I guess I have to start with adding a virtual NIC for each Kube node into the 10.10.10.0/24 network so ther can communicate with the Proxmox Ceph Mons - am I on the right track here or is there a better way to do this?
     
  2. breakaway9000

    breakaway9000 Member

    Joined:
    Dec 20, 2015
    Messages:
    49
    Likes Received:
    4
    Turns out that is not the right way to do it. The right way is to update the config so that the OSDs are running on 10.10.10.0/24 network (dedicated 10gbps) and put the monitors/managers on your 172.16.254.0/24 (or whatever) network.

    In the below scenario, I am setting up a "Public" network for my ceph in the 172.16.254.0/24 subnet (with the monitor IPs changing from 10.10.10.1-3 to 172.16.254.50-52).

    Below is a quick-n-nasty way to do this. Note that this requires cluster downtime.

    First, export the ceph monitor map to a file ("MONMAP_FILE" in this case).

    Code:
    ceph mon getmap -o MONMAP_FILE
    Now, check what's inside it.

    Code:
    monmaptool --print MONMAP_FILE
    Now, delete the old hosts & add the new hosts with the correct IP.

    Code:
    monmaptool --rm proxmox-node01 --rm proxmox-node02 --rm proxmox-node03
    monmaptool --add proxmox-node01 172.16.254.50:6789 --add proxmox-node02 172.16.254.51:6789 --add proxmox-node03 172.16.254.52:6789 MONMAP_FILE
    Check everything - make sure you got your IPs & hostnames correct.

    Then, move the file to /etc/pve - corosync will now put it on all your hosts. You don't have to do it this way but I found it to be the easiest way to shift stuff between nodes quickly.

    Code:
    mv MONMAP_FILE /etc/pve
    Now connect to your proxmox hosts, stop all VMs & CTs. Then, navigate to "ceph" and then to "Monitors", then stop all monitors. Now its time to inject the new monmap file into each host.

    Run this command on each host - modify as needed of course

    Code:
    ceph-mon -i <hostname> --inject-monmap /path/to/MONMAP_FILE
    For eg, I ran the following 3 commands

    Code:
    ceph-mon -i proxmox-node01 --inject-monmap /etc/pve/MONMAP_FILE
    ceph-mon -i proxmox-node02 --inject-monmap /etc/pve/MONMAP_FILE
    ceph-mon -i proxmox-node03 --inject-monmap /etc/pve/MONMAP_FILE
    Once that's done, you need to edit /etc/pve/ceph.conf to include your new IP addressing. I have posted my FULL ceph.conf for reference.

    Code:
    [global]
             auth client required = none
             auth cluster required = none
             auth service required = none
             fsid = sdf-287b-46ff-a302-sdfasdf
             keyring = /etc/pve/priv/$cluster.$name.keyring
             mon allow pool delete = true
             osd journal size = 5120
             osd pool default min size = 2
             osd pool default size = 3
             public network = 172.16.254.0/24
             cluster network = 10.10.10.0/24
    
    [osd]
             keyring = /var/lib/ceph/osd/ceph-$id/keyring
    
    [mon.proxmox-node01]
             host = proxmox-node01
             mon addr = 172.16.254.50:6789
    
    [mon.proxmox-node02]
             host = proxmox-node02
             mon addr = 172.16.254.51:6789
    
    [mon.aproxmox-node03]
             host = proxmox-node03
             mon addr = 172.16.254.52:6789
    At this point, you're supposed to be able to restart the monitors and then the OSDs one by one and everything should just work (TM) but I couldn't get this working. I rebooted all 3 of my hosts - everything now functioning as normal.

    I'd appreciate if anyone can chime in on why it doesn't work without a reboot.

    References:
    [1] http://docs.ceph.com/docs/mimic/rados/configuration/network-config-ref/
    [2] http://docs.ceph.com/docs/mimic/rados/operations/operating/
    [3] http://mohankri.weebly.com/my-interest/ceph-monitor-ip-address-changed
    [4] https://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/
    [5] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-July/002702.html
    [6] http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice