Ceph dashboard RGW problem

R

RickardNilsson

New Member
Nov 25, 2023
4
0
1
Hi,
I have a proxmox cluster consisting of five nodes. I have tried to configuring an RGW instance and then using the Ceph dashboard to configure the buckets and users but this fails.
The command that I supposed to be used for this,
Code: 
ceph dashboard set-rgw-credentials
fails with
Code: 
Error EINVAL: No RGW credentials found, please consult the documentation on how to enable RGW for the dashboard.
I instead created a dashboard user manually using the rados-admin command and generated credentials and configured the dashboard to use them, so far so good.
When I open the Object Gateway page in the dashboard I get the following error:
Code: 
Error connecting to Object Gateway: RGW REST API failed request with status code 404 (b'{"Code":"NoSuchKey","BucketName":"dashboard","RequestId":"tx00000f95f36044e2' b'dbcba-0066ce0da5-2975f046-sweden-uppsala","HostId":"2975f046-sweden-uppsala-' b'sweden"}')
The error logs in the rgw look like:
Code: 
Tue, 27 Aug 2024 17:32:21 GMT
/dashboard/metadata/user
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj server signature=ACNXf+IOmDW05ZaimE5ATlpdL60=
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj client signature=ACNXf+IOmDW05ZaimE5ATlpdL60=
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj compare=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj rgw::auth::s3::LocalEngine granted access
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj normalizing buckets and tenants
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s->object=metadata/user s->bucket=dashboard
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj init permissions
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: rctx=0x73d91820d1c0 obj=sweden-uppsala.rgw.meta:root:dashboard state=0x73d618003a20 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+root+dashboard : hit (requested=0x16, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: s->obj_tag was set empty
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+root+dashboard : hit (requested=0x11, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: rctx=0x73d91820ccb8 obj=sweden-uppsala.rgw.meta:users.uid:dashboard state=0x73d618003a20 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+users.uid+dashboard : hit (requested=0x16, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: s->obj_tag was set empty
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj Read xattr: user.rgw.idtag
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+users.uid+dashboard : hit (requested=0x13, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj recalculating target
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj reading permissions
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_obj_state: rctx=0x73d91820da40 obj=dashboard:metadata/user state=0x73d618016e68 s->prefetch_data=1
2024-08-27T17:32:21.483+0000 73d8716006c0 15 req 17969140430350957754 0.000999994s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2024-08-27T17:32:21.483+0000 73d8716006c0 10 req 17969140430350957754 0.000999994s s3:get_obj read_permissions on :dashboard[738f60e1-675d-4ceb-8e3a-fdc6c3f97a23.695296747.1])::dashboard[738f60e1-675d-4ceb-8e3a-fdc6c3f97a23.695296747.1]):metadata/user only_bucket=0 ret=-2
2024-08-27T17:32:21.483+0000 73d8716006c0  1 req 17969140430350957754 0.000999994s op->ERRORHANDLER: err_no=-2 new_err_no=-2
2024-08-27T17:32:21.483+0000 73d8716006c0 20 req 17969140430350957754 0.000999994s get_system_obj_state: rctx=0x73d91820d760 obj=sweden-uppsala.rgw.log:script.postrequest. state=0x73d668001520 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d8716006c0 10 req 17969140430350957754 0.000999994s cache get: name=sweden-uppsala.rgw.log++script.postrequest. : hit (negative entry)
2024-08-27T17:32:21.483+0000 73d8716006c0  2 req 17969140430350957754 0.000999994s s3:get_obj op status=0
2024-08-27T17:32:21.483+0000 73d8716006c0  2 req 17969140430350957754 0.000999994s s3:get_obj http status=404
2024-08-27T17:32:21.483+0000 73d8716006c0  1 ====== req done req=0x73d91820e700 op status=0 http_status=404 latency=0.000999994s ======
2024-08-27T17:32:21.483+0000 73d8716006c0  1 beast: 0x73d91820e700: 192.168.1.8 - dashboard [27/Aug/2024:17:32:21.482 +0000] "GET /dashboard/metadata/user?myself HTTP/1.1" 404 160 - "python-requests/2.28.1" - latency=0.000999994s

Ceph version: 18.2.2

Has anyone faced a similar issue and knows how to solve it?
 
Last edited:
I run into the same problem today and found your issue here. This was the only finding I had.

With help of my colleagues we found out the issue. The system flag for your dashboard user seems to be missing.

Please create dashboard user like this:
Code: 
radosgw-admin user create --uid=dashboard --display-name=dashboard --system
or modify existing:
Code: 
radosgw-admin user modify --uid=dashboard --system --caps=""

I re-created it without setting any caps (did before) and now it's working.

Other problem I run into, I set mgr/dashboard/RGW_API_ADMIN_RESOURCE to dashboard earlier, this needs to be set to admin or deleted. Maybe helpful for other people finding this post.
 
malus said:
I run into the same problem today and found your issue here. This was the only finding I had.

With help of my colleagues we found out the issue. The system flag for your dashboard user seems to be missing.

Please create dashboard user like this:
Code: 
radosgw-admin user create --uid=dashboard --display-name=dashboard --system
or modify existing:
Code: 
radosgw-admin user modify --uid=dashboard --system --caps=""

I re-created it without setting any caps (did before) and now it's working.

Other problem I run into, I set mgr/dashboard/RGW_API_ADMIN_RESOURCE to dashboard earlier, this needs to be set to admin or deleted. Maybe helpful for other people finding this post.
Click to expand...
This solved the issue for me, thanks

This is the error code I was getting for future searchability:
Code: 
RGW REST API failed request with status code 403 '{"Code":"AccessDenied","RequestId":"tx00000000000000000036d-005dfa0ba1-11e8-my-store","HostId":"11e8-my-store-my-store"}'
 
daltonpearson said:
This solved the issue for me, thanks

This is the error code I was getting for future searchability:
Code: 
RGW REST API failed request with status code 403 '{"Code":"AccessDenied","RequestId":"tx00000000000000000036d-005dfa0ba1-11e8-my-store","HostId":"11e8-my-store-my-store"}'
Click to expand...
hey! did you solve your problem with AccessDenied?
 
Hi,

I'm still stuck att the 404 error unfortunately, I wonder what it tries to read from the bucket that causes a 404 error...

//Rickard
 
Hi, having the same issues. After much fiddling around have managed to get it to give me the 403 Error...
RGW REST API failed GET req status: 403
2025-04-11T15:38:01.896+0200 7f5f2c7666c0 0 [dashboard ERROR rgw_client] RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","Message":"","RequestId":"tx00000ba305b8b10d'
b'56359-0067f91b39-27432108-default","HostId":"27432108-default-default"}')

In my case the error came from it using the wrong DNS Name.

After fixing that i now get this 404 Error:
Code: 
2025-04-11T16:07:23.903+0200 7f5f297486c0  0 [dashboard ERROR rgw_client] RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","BucketName":"dashboard","RequestId":"tx'
 b'0000038506725ec07000b-0067f9221b-27476776-default","HostId":"27476776-defaul'
 b't-default"}')
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 463, in __init__
    self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper
    return func(
           ^^^^^
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 498, in _get_user_id
    response = request()
               ^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 324, in __call__
    resp = self.rest_client.do_request(method, self._gen_path(), params,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 422, in do_request
    raise RequestException(
dashboard.rest_client.RequestException: RGW REST API failed request with status code 404
(b'{"Code":"NoSuchBucket","Message":"","BucketName":"dashboard","RequestId":"tx'
 b'0000038506725ec07000b-0067f9221b-27476776-default","HostId":"27476776-defaul'
 b't-default"}')
 
You must log in or register to reply here.
Top Bottom