Ceph dashboard RGW problem

[RickardNilsson]

Hi,
I have a proxmox cluster consisting of five nodes. I have tried to configuring an RGW instance and then using the Ceph dashboard to configure the buckets and users but this fails.
The command that I supposed to be used for this,

ceph dashboard set-rgw-credentials

fails with

Error EINVAL: No RGW credentials found, please consult the documentation on how to enable RGW for the dashboard.

I instead created a dashboard user manually using the rados-admin command and generated credentials and configured the dashboard to use them, so far so good.
When I open the Object Gateway page in the dashboard I get the following error:

Error connecting to Object Gateway: RGW REST API failed request with status code 404 (b'{"Code":"NoSuchKey","BucketName":"dashboard","RequestId":"tx00000f95f36044e2' b'dbcba-0066ce0da5-2975f046-sweden-uppsala","HostId":"2975f046-sweden-uppsala-' b'sweden"}')

The error logs in the rgw look like:

Tue, 27 Aug 2024 17:32:21 GMT
/dashboard/metadata/user
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj server signature=ACNXf+IOmDW05ZaimE5ATlpdL60=
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj client signature=ACNXf+IOmDW05ZaimE5ATlpdL60=
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj compare=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj rgw::auth::s3::LocalEngine granted access
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj normalizing buckets and tenants
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s->object=metadata/user s->bucket=dashboard
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj init permissions
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: rctx=0x73d91820d1c0 obj=sweden-uppsala.rgw.meta:root:dashboard state=0x73d618003a20 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+root+dashboard : hit (requested=0x16, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: s->obj_tag was set empty
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+root+dashboard : hit (requested=0x11, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 15 req 17969140430350957754 0.000999994s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: rctx=0x73d91820ccb8 obj=sweden-uppsala.rgw.meta:users.uid:dashboard state=0x73d618003a20 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+users.uid+dashboard : hit (requested=0x16, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_system_obj_state: s->obj_tag was set empty
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj Read xattr: user.rgw.idtag
2024-08-27T17:32:21.483+0000 73d6bd2006c0 10 req 17969140430350957754 0.000999994s s3:get_obj cache get: name=sweden-uppsala.rgw.meta+users.uid+dashboard : hit (requested=0x13, cached=0x17)
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj recalculating target
2024-08-27T17:32:21.483+0000 73d6bd2006c0  2 req 17969140430350957754 0.000999994s s3:get_obj reading permissions
2024-08-27T17:32:21.483+0000 73d6bd2006c0 20 req 17969140430350957754 0.000999994s s3:get_obj get_obj_state: rctx=0x73d91820da40 obj=dashboard:metadata/user state=0x73d618016e68 s->prefetch_data=1
2024-08-27T17:32:21.483+0000 73d8716006c0 15 req 17969140430350957754 0.000999994s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>dashboard</ID><DisplayName>Dashboard</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2024-08-27T17:32:21.483+0000 73d8716006c0 10 req 17969140430350957754 0.000999994s s3:get_obj read_permissions on :dashboard[738f60e1-675d-4ceb-8e3a-fdc6c3f97a23.695296747.1])::dashboard[738f60e1-675d-4ceb-8e3a-fdc6c3f97a23.695296747.1]):metadata/user only_bucket=0 ret=-2
2024-08-27T17:32:21.483+0000 73d8716006c0  1 req 17969140430350957754 0.000999994s op->ERRORHANDLER: err_no=-2 new_err_no=-2
2024-08-27T17:32:21.483+0000 73d8716006c0 20 req 17969140430350957754 0.000999994s get_system_obj_state: rctx=0x73d91820d760 obj=sweden-uppsala.rgw.log:script.postrequest. state=0x73d668001520 s->prefetch_data=0
2024-08-27T17:32:21.483+0000 73d8716006c0 10 req 17969140430350957754 0.000999994s cache get: name=sweden-uppsala.rgw.log++script.postrequest. : hit (negative entry)
2024-08-27T17:32:21.483+0000 73d8716006c0  2 req 17969140430350957754 0.000999994s s3:get_obj op status=0
2024-08-27T17:32:21.483+0000 73d8716006c0  2 req 17969140430350957754 0.000999994s s3:get_obj http status=404
2024-08-27T17:32:21.483+0000 73d8716006c0  1 ====== req done req=0x73d91820e700 op status=0 http_status=404 latency=0.000999994s ======
2024-08-27T17:32:21.483+0000 73d8716006c0  1 beast: 0x73d91820e700: 192.168.1.8 - dashboard [27/Aug/2024:17:32:21.482 +0000] "GET /dashboard/metadata/user?myself HTTP/1.1" 404 160 - "python-requests/2.28.1" - latency=0.000999994s

Ceph version: 18.2.2

Has anyone faced a similar issue and knows how to solve it?

 

[malus]

I run into the same problem today and found your issue here. This was the only finding I had.

With help of my colleagues we found out the issue. The system flag for your dashboard user seems to be missing.

Please create dashboard user like this:

radosgw-admin user create --uid=dashboard --display-name=dashboard --system

or modify existing:

radosgw-admin user modify --uid=dashboard --system --caps=""

I re-created it without setting any caps (did before) and now it's working.

Other problem I run into, I set mgr/dashboard/RGW_API_ADMIN_RESOURCE to dashboard earlier, this needs to be set to admin or deleted. Maybe helpful for other people finding this post.