Hello, community.
Is there a way to activate and use Ceph Object Storage in a Proxmox Ceph Cluster?
Thanks
Is there a way to activate and use Ceph Object Storage in a Proxmox Ceph Cluster?
Thanks
Activate Ceph Object Storage
- Thread starter Dan Nicolae
- Start date
Please check out the docs.
https://pve.proxmox.com/pve-docs/chapter-pveceph.html
And go through the post in this thread.
https://forum.proxmox.com/threads/ceph-raw-usage-grows-by-itself.38395/#post-189842
https://pve.proxmox.com/pve-docs/chapter-pveceph.html
And go through the post in this thread.
https://forum.proxmox.com/threads/ceph-raw-usage-grows-by-itself.38395/#post-189842
That's block storage. Ceph Luminous is capable to function as an object storage, via web API (RADOS Gateway), similar to Amazon S3, Swift...
Oficial Ceph tutorial to configure block storage can be found here and here.
Is there a way, to activate it in a Proxmox Ceph Cluster and transform a Proxmox Ceph node as an gateway?
Unfortunately pveceph install radosgw is not working.
Thanks!
Oficial Ceph tutorial to configure block storage can be found here and here.
Is there a way, to activate it in a Proxmox Ceph Cluster and transform a Proxmox Ceph node as an gateway?
Unfortunately pveceph install radosgw is not working.
Thanks!
Last edited:
So, one of the solutions is to create a non-proxmox ceph cluster and add it to the proxmox compute node as an external rbd.
That or you configure the additional service on a pve+ceph node. But as stated above, we are not supporting that configuration. Also AFAIK the RGW should reside on a different server then MONs or OSDs.
Thanks for the answers. Unofficial, any ideea how to install ceph radosgw on an existing proxmox ceph node?
PS: I know that you are not supporting that configuration.
PS: I know that you are not supporting that configuration.
We wrote a wrapper around the ceph tools. The ceph way of installing the RGW should still work.
I found a nice tutorial called Adding S3 capabilities to Proxmox, that I followed step by step and all works fine.
Proxmox Virtualization Environment (VE) is an outstanding virtualization platform. It has a number of great features that you don't get in many other enterprise platforms. One of these features is Ceph support, including the ability to run Ceph on the Proxmox nodes themselves. In addition to using Ceph for my VM's and Containers, I wanted to take advantage of the Ceph Object Gateway to get S3 compatible storage for my websites.
General disclaimer: I am fairly new to Ceph and Ceph Object Gateway, so these instructions may not be the optimal way to do this, but it worked for me, so I though I'd share.
I mostly followed the instructions from the main Ceph site, but it was somewhat confusing because they referred to installing Apache and FCGI in some places, but in others they mention that Ceph uses "Civetweb". There is also mention of using ceph-deploy, but I knew that Proxmox uses it's own pveceph tools. So, not wanting to affect my main Proxmox nodes too much, I decided on my first cut to install a dual NIC VM and put one on the same VLAN as my storage network, and the other on the PVE VLAN. It went well enough, and only required one additional package, so I decided to go ahead and install directly on the Proxmox nodes.
My Proxmox environment consists of 3 nodes: pve1, pve2, and pve3, and I wanted to run the Gateway on all three nodes for High Availabilty (I'm running HAProxy in front of these for SSL termination, HA and load balancing).
I ran the following commands from the pve1 node, but it could have been done from any of the nodes.
First I created the keyring to store the keys:
root@pve1:~# ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
Next, I generated the keys and added them to the keyring:
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve1 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve2 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve3 --gen-key
And then I added the proper capabilities:
root@pve1:~# ceph-authtool -n client.radosgw.pve1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
Finally, I add the keys to the cluster:
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve1 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve2 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve3 -i /etc/ceph/ceph.client.radosgw.keyring
I also copied the keyring into the Proxmox ClusterFS so that it'd be available on all nodes. Note: I might have been able to generate the key directly in the /etc/pve/priv folder, and saved this step.
root@pve1:~# cp /etc/ceph/ceph.client.radosgw.keyring /etc/pve/priv
Add the following lines to /etc/ceph/ceph.conf:
[client.radosgw.pve1]
host = pve1
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.example.net
[client.radosgw.pve2]
host = pve2
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.example.net
[client.radosgw.pve3]
host = pve3
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.rados.$host.log
rgw_dns_name = s3.example.net
Here again, I think there's room for optimization. It's my understanding that multiple [client] sections can be combined, so everything below the host line could potentially be merged into a single section to eliminate repetition.
At this point it was time to log into each of the nodes and add the proper packages:
root@pve1:~# apt install radosgw
And then fire it up:
root@pve1:~# service radosgw start
If all goes well, RADOSGW will create some default pools for you (see below), and you should be able to visit any of your nodes on port 7480 (e.g. http://pve1.example.net:7480) and you should see something like this:
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>anonymous</ID>
<DisplayName/>
</Owner>
<Buckets/>
</ListAllMyBucketsResult>
If not, you can follow your logs to troubleshoot:
root@pve1:~# tail -f /var/log/ceph/client.rados.pve1.log
I was getting warnings on my Ceph cluster that the application hadn't been enabled on pools, so I ran the following:
root@pve1:~# ceph osd pool application enable .rgw.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.control rgw
root@pve1:~# ceph osd pool application enable default.rgw.data.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.gc rgw
root@pve1:~# ceph osd pool application enable default.rgw.log rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.uid rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.email rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.keys rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.index rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.data rgw
root@pve1:~# ceph osd pool application enable default.rgw.lc rgw
Note: some of these pools showed up only when I needed them, such as creating a user, so I may need to go back and rerun this command with any newly created pools
So now you can setup your first user:
root@pve1:~# radosgw-admin user create --uid=testuser --display-name="Test User" --email=test.user@example.net
That's it for configuration on the servers. If you plan to expose these as I did through HAProxy, don't forget to add a wildcard entry for your domain in DNS: *.s3.example.net, so that your buckets will resolve. I also ended up purchasing a wildcard SSL certificate that I loaded onto HAProxy for SSL.
Proxmox Virtualization Environment (VE) is an outstanding virtualization platform. It has a number of great features that you don't get in many other enterprise platforms. One of these features is Ceph support, including the ability to run Ceph on the Proxmox nodes themselves. In addition to using Ceph for my VM's and Containers, I wanted to take advantage of the Ceph Object Gateway to get S3 compatible storage for my websites.
General disclaimer: I am fairly new to Ceph and Ceph Object Gateway, so these instructions may not be the optimal way to do this, but it worked for me, so I though I'd share.
I mostly followed the instructions from the main Ceph site, but it was somewhat confusing because they referred to installing Apache and FCGI in some places, but in others they mention that Ceph uses "Civetweb". There is also mention of using ceph-deploy, but I knew that Proxmox uses it's own pveceph tools. So, not wanting to affect my main Proxmox nodes too much, I decided on my first cut to install a dual NIC VM and put one on the same VLAN as my storage network, and the other on the PVE VLAN. It went well enough, and only required one additional package, so I decided to go ahead and install directly on the Proxmox nodes.
My Proxmox environment consists of 3 nodes: pve1, pve2, and pve3, and I wanted to run the Gateway on all three nodes for High Availabilty (I'm running HAProxy in front of these for SSL termination, HA and load balancing).
I ran the following commands from the pve1 node, but it could have been done from any of the nodes.
First I created the keyring to store the keys:
root@pve1:~# ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring
Next, I generated the keys and added them to the keyring:
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve1 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve2 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve3 --gen-key
And then I added the proper capabilities:
root@pve1:~# ceph-authtool -n client.radosgw.pve1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
Finally, I add the keys to the cluster:
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve1 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve2 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve3 -i /etc/ceph/ceph.client.radosgw.keyring
I also copied the keyring into the Proxmox ClusterFS so that it'd be available on all nodes. Note: I might have been able to generate the key directly in the /etc/pve/priv folder, and saved this step.
root@pve1:~# cp /etc/ceph/ceph.client.radosgw.keyring /etc/pve/priv
Add the following lines to /etc/ceph/ceph.conf:
[client.radosgw.pve1]
host = pve1
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.example.net
[client.radosgw.pve2]
host = pve2
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.radosgw.$host.log
rgw_dns_name = s3.example.net
[client.radosgw.pve3]
host = pve3
keyring = /etc/pve/priv/ceph.client.radosgw.keyring
log file = /var/log/ceph/client.rados.$host.log
rgw_dns_name = s3.example.net
Here again, I think there's room for optimization. It's my understanding that multiple [client] sections can be combined, so everything below the host line could potentially be merged into a single section to eliminate repetition.
At this point it was time to log into each of the nodes and add the proper packages:
root@pve1:~# apt install radosgw
And then fire it up:
root@pve1:~# service radosgw start
If all goes well, RADOSGW will create some default pools for you (see below), and you should be able to visit any of your nodes on port 7480 (e.g. http://pve1.example.net:7480) and you should see something like this:
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>anonymous</ID>
<DisplayName/>
</Owner>
<Buckets/>
</ListAllMyBucketsResult>
If not, you can follow your logs to troubleshoot:
root@pve1:~# tail -f /var/log/ceph/client.rados.pve1.log
I was getting warnings on my Ceph cluster that the application hadn't been enabled on pools, so I ran the following:
root@pve1:~# ceph osd pool application enable .rgw.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.control rgw
root@pve1:~# ceph osd pool application enable default.rgw.data.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.gc rgw
root@pve1:~# ceph osd pool application enable default.rgw.log rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.uid rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.email rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.keys rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.index rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.data rgw
root@pve1:~# ceph osd pool application enable default.rgw.lc rgw
Note: some of these pools showed up only when I needed them, such as creating a user, so I may need to go back and rerun this command with any newly created pools
So now you can setup your first user:
root@pve1:~# radosgw-admin user create --uid=testuser --display-name="Test User" --email=test.user@example.net
That's it for configuration on the servers. If you plan to expose these as I did through HAProxy, don't forget to add a wildcard entry for your domain in DNS: *.s3.example.net, so that your buckets will resolve. I also ended up purchasing a wildcard SSL certificate that I loaded onto HAProxy for SSL.
How did you get this to work? I'm facing dependency issues due to custom *-pve packages from Proxmox:
[user@proxmox ~]# sudo apt-get install radosgw
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
radosgw : Depends: ceph-common (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
Depends: librgw2 (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
E: Unable to correct problems, you have held broken packages.
PS. I've added the officiel ceph debian repo due to radosgw not being in the proxmox repo.
>> I found a nice tutorial called <Adding S3 capabilities to Proxmox>, that I followed step by step and all works fine. <<
[user@proxmox ~]# sudo apt-get install radosgw
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
radosgw : Depends: ceph-common (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
Depends: librgw2 (= 12.2.10-1~bpo90+1) but 12.2.10-pve1 is to be installed
E: Unable to correct problems, you have held broken packages.
PS. I've added the officiel ceph debian repo due to radosgw not being in the proxmox repo.
>> I found a nice tutorial called <Adding S3 capabilities to Proxmox>, that I followed step by step and all works fine. <<
Last edited:
Maybe it didn't come up during apt install due to it being 12.2.8 for radosgw and 12.2.10 for the rest of ceph?
I already did the containerized setup and it works as an alternative... may actually be better this way so i can control the impact on the host.
We create all packages from upstream, with additional patches for backported fixes/features.
Everyone. Thank you for this thread discuss!
I followed the instructions to install S3. It works great. However, when I reboot, it stops working.
I would have to remove and reinstall radosgw. Any ideas on how to keep radosgw working after reboot?
Thanks in advance.
I followed the instructions to install S3. It works great. However, when I reboot, it stops working.
I would have to remove and reinstall radosgw. Any ideas on how to keep radosgw working after reboot?
Thanks in advance.
Based on the guide above its enabled via pool setting, which clear every reboot. You should be able to re-enable it by running those settings again.
Thanks for the quick reply Paradox55. My next question is how to make it permanent? Is this something that ProxMox will not suppor?
My goal is to use Poxmox+Ceph as object storage.
Thank you.
What does proxmox have to do with this? It's ceph.
Configure your cluster correctly and it will be fine.