Activate Ceph Object Storage

Discussion in 'Proxmox VE: Installation and configuration' started by Dan Nicolae, Feb 16, 2018.

  1. Dan Nicolae

    Dan Nicolae Member

    Joined:
    Apr 27, 2016
    Messages:
    69
    Likes Received:
    2
    Hello, community.

    Is there a way to activate and use Ceph Object Storage in a Proxmox Ceph Cluster?

    Thanks
     
  2. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Dan Nicolae

    Dan Nicolae Member

    Joined:
    Apr 27, 2016
    Messages:
    69
    Likes Received:
    2
    That's block storage. Ceph Luminous is capable to function as an object storage, via web API (RADOS Gateway), similar to Amazon S3, Swift...
    Oficial Ceph tutorial to configure block storage can be found here and here.

    Is there a way, to activate it in a Proxmox Ceph Cluster and transform a Proxmox Ceph node as an gateway?

    Unfortunately pveceph install radosgw is not working. :)

    Thanks!
     
    #3 Dan Nicolae, Feb 22, 2018
    Last edited: Feb 22, 2018
  4. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    Ceph is capable of many things, that said, while you can configure ceph to your liking (using ceph tools), we are not supporting RGW.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. Dan Nicolae

    Dan Nicolae Member

    Joined:
    Apr 27, 2016
    Messages:
    69
    Likes Received:
    2
    So, one of the solutions is to create a non-proxmox ceph cluster and add it to the proxmox compute node as an external rbd.
     
  6. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    That or you configure the additional service on a pve+ceph node. But as stated above, we are not supporting that configuration. Also AFAIK the RGW should reside on a different server then MONs or OSDs.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Dan Nicolae likes this.
  7. Dan Nicolae

    Dan Nicolae Member

    Joined:
    Apr 27, 2016
    Messages:
    69
    Likes Received:
    2
    Thanks for the answers. Unofficial, any ideea how to install ceph radosgw on an existing proxmox ceph node?

    PS: I know that you are not supporting that configuration. ;)
     
  8. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    We wrote a wrapper around the ceph tools. The ceph way of installing the RGW should still work.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Dan Nicolae likes this.
  9. Dan Nicolae

    Dan Nicolae Member

    Joined:
    Apr 27, 2016
    Messages:
    69
    Likes Received:
    2
    I found a nice tutorial called Adding S3 capabilities to Proxmox, that I followed step by step and all works fine.


    Proxmox Virtualization Environment (VE) is an outstanding virtualization platform. It has a number of great features that you don't get in many other enterprise platforms. One of these features is Ceph support, including the ability to run Ceph on the Proxmox nodes themselves. In addition to using Ceph for my VM's and Containers, I wanted to take advantage of the Ceph Object Gateway to get S3 compatible storage for my websites.

    General disclaimer: I am fairly new to Ceph and Ceph Object Gateway, so these instructions may not be the optimal way to do this, but it worked for me, so I though I'd share.

    I mostly followed the instructions from the main Ceph site, but it was somewhat confusing because they referred to installing Apache and FCGI in some places, but in others they mention that Ceph uses "Civetweb". There is also mention of using ceph-deploy, but I knew that Proxmox uses it's own pveceph tools. So, not wanting to affect my main Proxmox nodes too much, I decided on my first cut to install a dual NIC VM and put one on the same VLAN as my storage network, and the other on the PVE VLAN. It went well enough, and only required one additional package, so I decided to go ahead and install directly on the Proxmox nodes.

    My Proxmox environment consists of 3 nodes: pve1, pve2, and pve3, and I wanted to run the Gateway on all three nodes for High Availabilty (I'm running HAProxy in front of these for SSL termination, HA and load balancing).

    I ran the following commands from the pve1 node, but it could have been done from any of the nodes.

    First I created the keyring to store the keys:

    root@pve1:~# ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
    Next, I generated the keys and added them to the keyring:

    root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve1 --gen-key
    root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve2 --gen-key
    root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve3 --gen-key
    And then I added the proper capabilities:

    root@pve1:~# ceph-authtool -n client.radosgw.pve1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
    root@pve1:~# ceph-authtool -n client.radosgw.pve2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
    root@pve1:~# ceph-authtool -n client.radosgw.pve3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
    Finally, I add the keys to the cluster:

    root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve1 -i /etc/ceph/ceph.client.radosgw.keyring
    root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve2 -i /etc/ceph/ceph.client.radosgw.keyring
    root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve3 -i /etc/ceph/ceph.client.radosgw.keyring
    I also copied the keyring into the Proxmox ClusterFS so that it'd be available on all nodes. Note: I might have been able to generate the key directly in the /etc/pve/priv folder, and saved this step.

    root@pve1:~# cp /etc/ceph/ceph.client.radosgw.keyring /etc/pve/priv
    Add the following lines to /etc/ceph/ceph.conf:

    [client.radosgw.pve1]
    host = pve1
    keyring = /etc/pve/priv/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.radosgw.$host.log
    rgw_dns_name = s3.example.net

    [client.radosgw.pve2]
    host = pve2
    keyring = /etc/pve/priv/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.radosgw.$host.log
    rgw_dns_name = s3.example.net

    [client.radosgw.pve3]
    host = pve3
    keyring = /etc/pve/priv/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.rados.$host.log
    rgw_dns_name = s3.example.net

    Here again, I think there's room for optimization. It's my understanding that multiple [client] sections can be combined, so everything below the host line could potentially be merged into a single section to eliminate repetition.

    At this point it was time to log into each of the nodes and add the proper packages:

    root@pve1:~# apt install radosgw
    And then fire it up:

    root@pve1:~# service radosgw start
    If all goes well, RADOSGW will create some default pools for you (see below), and you should be able to visit any of your nodes on port 7480 (e.g. http://pve1.example.net:7480) and you should see something like this:

    <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
    <ID>anonymous</ID>
    <DisplayName/>
    </Owner>
    <Buckets/>
    </ListAllMyBucketsResult>
    If not, you can follow your logs to troubleshoot:

    root@pve1:~# tail -f /var/log/ceph/client.rados.pve1.log
    I was getting warnings on my Ceph cluster that the application hadn't been enabled on pools, so I ran the following:

    root@pve1:~# ceph osd pool application enable .rgw.root rgw
    root@pve1:~# ceph osd pool application enable default.rgw.control rgw
    root@pve1:~# ceph osd pool application enable default.rgw.data.root rgw
    root@pve1:~# ceph osd pool application enable default.rgw.gc rgw
    root@pve1:~# ceph osd pool application enable default.rgw.log rgw
    root@pve1:~# ceph osd pool application enable default.rgw.users.uid rgw
    root@pve1:~# ceph osd pool application enable default.rgw.users.email rgw
    root@pve1:~# ceph osd pool application enable default.rgw.users.keys rgw
    root@pve1:~# ceph osd pool application enable default.rgw.buckets.index rgw
    root@pve1:~# ceph osd pool application enable default.rgw.buckets.data rgw
    root@pve1:~# ceph osd pool application enable default.rgw.lc rgw

    Note: some of these pools showed up only when I needed them, such as creating a user, so I may need to go back and rerun this command with any newly created pools

    So now you can setup your first user:

    root@pve1:~# radosgw-admin user create --uid=testuser --display-name="Test User" --email=test.user@example.net
    That's it for configuration on the servers. If you plan to expose these as I did through HAProxy, don't forget to add a wildcard entry for your domain in DNS: *.s3.example.net, so that your buckets will resolve. I also ended up purchasing a wildcard SSL certificate that I loaded onto HAProxy for SSL.
     
  10. Robert.H

    Robert.H New Member

    Joined:
    Jan 11, 2019
    Messages:
    5
    Likes Received:
    0
    How did you get this to work? I'm facing dependency issues due to custom *-pve packages from Proxmox:

    [user@proxmox ~]# sudo apt-get install radosgw
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    distribution that some required packages have not yet been created
    or been moved out of Incoming.
    The following information may help to resolve the situation:

    The following packages have unmet dependencies:
    radosgw : Depends: ceph-common (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
    Depends: librgw2 (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
    E: Unable to correct problems, you have held broken packages.

    PS. I've added the officiel ceph debian repo due to radosgw not being in the proxmox repo.

    >> I found a nice tutorial called <Adding S3 capabilities to Proxmox>, that I followed step by step and all works fine. <<
     
    #10 Robert.H, Jan 11, 2019
    Last edited: Jan 11, 2019
  11. Robert.H

    Robert.H New Member

    Joined:
    Jan 11, 2019
    Messages:
    5
    Likes Received:
    0
    I'm thinking that having the radosgw in a container might be a better idea... And goign from there, why isn't everything promox provides inside containers? hmm....
     
  12. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. Robert.H

    Robert.H New Member

    Joined:
    Jan 11, 2019
    Messages:
    5
    Likes Received:
    0
  14. Alwin

    Alwin Proxmox Staff Member
    Staff Member

    Joined:
    Aug 1, 2017
    Messages:
    2,093
    Likes Received:
    184
    We create all packages from upstream, with additional patches for backported fixes/features.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice