[SOLVED] Ceph Object RGW

H

harleydica

New Member
Jul 13, 2024
14
1
3
Hello, I created a ceph and added a ceph dashboard, then I want to create an object bucket, but there is an error as shown below. Maybe someone can help me?
1737416158030.png
 
I'm trying to get RGW working on Proxmox too. If anyone else has any insights or resources they can link to, I'd appreciate it.

Stuff I've tried
Here's some of what I've learned and tried. Some of this was sourced from other posts in the forum. Other info came from older guides I found.
  • Proxmox doesn't install RadosGW by default. You can install it manually from apt with the radosgwpackage.
    • You'll need to manually make a symlink to the SystemD service unit to start it on the machines you want to run RGW
    • You need to set up a keyring for each RGW client. I followed steps in the wiki for that.
  • Ceph doesn't have permissions to access /etc/pve/ceph.conf by default. To fix that, I added the secondary group www-data to user ceph.
  • The Proxmox distribution of ceph-mgr-dashboard doesn't include code for the rgw module. You can manually fetch these two files from source. You need to do this for each of your ceph-mgr instances, then enable the rgw module on one.
I've got an ansible playbook that does most of the steps above. Let me know if you want me to share it.

Also, Ceph Dashboard expects you to be using cephadmin/ceph orch to manage daemons in some random places. The Proxmox distribution of Ceph doesn't use cephadmin. It manages Ceph daemons in a different way.

Where I'm stuck
I have RGW running without errors, but the dashboard can't connect to it. When I try to access any of the 'Object' pages on the dashboard, the web browser reports "500 - Internal Server Error". I found the server logs on the active mgr:

Code: 
2025-01-24T11:32:25.534-0500 7df4e90006c0  0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=192.168.1.201, port=7480, ssl=False
2025-01-24T11:32:25.534-0500 7df4e90006c0  0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=192.168.1.203, port=7480, ssl=False
2025-01-24T11:32:25.535-0500 7df4e90006c0  0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=192.168.1.202, port=7480, ssl=False
2025-01-24T11:32:25.535-0500 7df4e90006c0  0 [dashboard INFO rgw_client] Configuring dashboard RGW credentials
2025-01-24T11:32:25.563-0500 7df4e90006c0  0 [dashboard ERROR rgw_client] Unable to list RGW realms: global_init: unable to open config file from search list /etc/ceph/ceph.conf

2025-01-24T11:32:25.618-0500 7df4e90006c0  0 [dashboard ERROR rgw_client] Unable to create rgw user "dashboard": global_init: unable to open config file from search list /etc/ceph/ceph.conf


2025-01-24T11:32:25.618-0500 7df4e90006c0  0 [dashboard ERROR rgw_client]
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 283, in configure_rgw_credentials
    assert access_key and secret_key
AssertionError
2025-01-24T11:32:25.619-0500 7df4e90006c0  0 [dashboard ERROR exception] Internal Server Error
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 283, in configure_rgw_credentials
    assert access_key and secret_key
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
    return handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
    ret = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 58, in status
    instance = RgwClient.admin_instance()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 404, in admin_instance
    return RgwClient.instance(daemon_name=daemon_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 351, in instance
    configure_rgw_credentials()
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 288, in configure_rgw_credentials
    raise NoCredentialsException
dashboard.services.rgw_client.NoCredentialsException: No RGW credentials found, please consult the documentation on how to enable RGW for the dashboard.
2025-01-24T11:32:25.619-0500 7df4e90006c0  0 [dashboard ERROR request] [::ffff:192.168.1.232:62290] [GET] [500] [0.088s] [cephdash] [513.0B] /ui-api/rgw/status
2025-01-24T11:32:25.619-0500 7df4e90006c0  0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented
 it from fulfilling the request.", "request_id": "7f43a57d-25c3-4a3b-b5ee-e3451113f2bd"}

                          ']
(I'm adding all this detail in cases someone else searches for by log output)

I followed steps from the official docs to create an RGW user and add the credentials to the dashboard. I have the RGW user:
Code: 
# radosgw-admin user list
[
    "dashboard",
    "benchmark"
]

When I try to add the credentials to the dashboard, I get the following error:
Code: 
# ceph dashboard set-rgw-credentials
Error EINVAL: No RGW credentials found, please consult the documentation on how to enable RGW for the dashboard.

It looks like I can also add them using ceph dashboard set-rgw-api-access-key + ceph dashboard set-rgw-api-secret-key. I'll try to figure that out next.

If anyone else has gotten the rgw module working on Proxmox in the Ceph Dashboard, please share. For a Ceph noob, it's taken a ton of effort to get this far.
 
I was able to set the RGW credentials for the dashboard! Here are the steps I followed:
  1. Run radosgw-admin user info --uid=dashboard | less to find the access and secret keys for the user I'd already created
  2. Save the access key value to access.key and the secret to secret.key.
  3. Set the credentials in the dashboard:
Code: 
# ceph dashboard set-rgw-api-access-key -i access.key
Option RGW_API_ACCESS_KEY updated

root@baldr:~# ceph dashboard set-rgw-api-secret-key -i secret.key
Option RGW_API_SECRET_KEY updated

The Object pages load for me now. But I'm still getting a few error.

The Object -> Overview page loads, but I get an error that pops up and says:

500 - Internal Server Error​

  • Unable to fetch realm list
  • Unable to fetch zone list
  • Unable to fetch zonegroup list
Click to expand...

Similarly, the Object -> Multi-site page gives the error:

500 - Internal Server Error​

Unable to fetch realm list
Click to expand...

Edit: Here's the command I used originally to create the 'dashboard' user. From another post in this forum, apparently the --system parameter is important.
Code: 
radosgw-admin user create --uid=dashboard-system-user --display-name="dashboard" --system
[CODE]
 
Last edited:
  • Like
Reactions: aufwiz and powderhorn
davad said:
I was able to set the RGW credentials for the dashboard! Here are the steps I followed:
  1. Run radosgw-admin user info --uid=dashboard | less to find the access and secret keys for the user I'd already created
  2. Save the access key value to access.key and the secret to secret.key.
  3. Set the credentials in the dashboard:
Code: 
# ceph dashboard set-rgw-api-access-key -i access.key
Option RGW_API_ACCESS_KEY updated

root@baldr:~# ceph dashboard set-rgw-api-secret-key -i secret.key
Option RGW_API_SECRET_KEY updated

The Object pages load for me now. But I'm still getting a few error.

The Object -> Overview page loads, but I get an error that pops up and says:


Similarly, the Object -> Multi-site page gives the error:


Edit: Here's the command I used originally to create the 'dashboard' user. From another post in this forum, apparently the --system parameter is important.
Code: 
radosgw-admin user create --uid=dashboard-system-user --display-name="dashboard" --system
[CODE]
Click to expand...
Thank you, I am currently able to use RGW well, but the problems that arise as you talked about are not a problem for me. Thank you for your response.
 
If we want virtualization via the CLI, we can use KVM. However, we choose Proxmox because it offers a GUI. Therefore, we expect that all features, including the configuration of the Object Gateway Service, should be accessible and manageable through the GUI or Dashboard. Leaving such configurations to undocumented or cumbersome CLI procedures is not what we expect from a GUI-based virtualization platform!
 
  • Like
Reactions: davad
@Drallas,

I was frustrated to find this too but there are some reasons why this is the case that I kind of understand.

1 - As much as I really want rgw, prox isn't obliged to support all of the features of ceph just because it supports parts of it. Proxmox is heavily made up of open source projects that it selectively exposes parts of. ZFS is a good example of this. Proxmox leans heavily on ZFS, LVM and CEPH and uses their features for a lot of things in the same way (some of which, like snapshots, are indistinguishable in the gui). But proxmox doesn't give you ANYTHING like all the capabilities of ZFS in either its gui or it's commandline tools. You can do all kinds of ZFS stuff on your host and its fine and should work great with prox (within logic), but you do have to learn the ZFS commandline interface.

2 - s3 isn't really a "virtualization" feature, and it is itself so feature rich that proxmox would need a lot of demand to support it. the other two Ceph interfaces, RBD and CephFS, proxmox exposes in some ways because they directly provide virtualization and/or HA functionality. S3 provides cloud compute features you'd expect from Digital Ocean, etc, (and I hope one day will be part of what proxmox supports) but Proxmox Virtual Environment isn't trying to be a storage solution. S3 is Simple Storage Service. Proxmox has some storage features but they are designed to provide storage for VMs, not for general purpose object storage or being an http gateway. thats more application oriented than just providing virtualization.

3 - proxmox uses pveceph which it developed before ceph orch and cephadm either existed or were robust and stable. these are the two tools that would make it easy to setup rgw from the commandline, and that interfere with ceph's dashboard functioning without errors in PVE - but they also are not compatible with pveceph and thats how proxmox orchistrates ceph. Basically as far as I can tell proxmox team was focused on making ceph usable for folks who want guis while ceph was younger and focused on being stable and adding features for more techncial users - eg. CERN. Prehaps the proxmox team could have crystal ball'd the appearance of cephadm and ceph orch and made pveceph as a wrapper for some code that they planned to donate to Ceph, that way it would have stayed integrated. But they couldn have known ceph would take thier code or whatever anyway so I cant really blame them there.

In any case I do hope that theres a way to make it easier to use rgw and cephs dashboard alongside proxmox's in the future as I think rgw is a fantastic tool and the ceph dashboard is a good compliment to PVE's dash. I'm using both in thier imperfectly integerated state atm. but if you really need rgw installed more easily, or really want a fully functional ceph dashobard, you can always setup proxmox and connect it to an external ceph cluster. (link at bottom of a 11m vid on how to connect). I bet you could even run ceph on the same machines as proxmox without a problem but you do have to learn ceph, just like you'd need to learn ZFS to get all of the fancy features there.


Configure Proxmox to use an external Ceph cluster​

https://www.youtube.com/watch?v=uBpug5f5XTs
 
  • Like
Reactions: herzkerl, tscret, aufwiz and 3 others
You must log in or register to reply here.
Top Bottom