Ceph Dashboard (RADOS GW management problem)

D

David Herselman

Renowned Member
Proxmox Subscriber
Jun 8, 2016
344
74
93
48
Johannesburg, South Africa
Hi,

We have been using the Ceph MGR Dashboard to successfully manage S3 buckets and user accounts since Octopus (https://forum.proxmox.com/threads/pve-6-3-with-ha-ceph-iscsi.81991/). This continued to work with us migrating to Ceph Pacific and exchanging civetweb for beast, although we annoyingly had to disable SSL verification as the Ceph Dashboard doesn't appear to read/honour the 'rgw_dns_name' value.

This worked up until Ceph Pacific 16.2.9, but broke after upgrading to Pacific 16.2.11. We recreated the RADOS gateway (aka RGW) user account and completely reconfigured everything with no success. We constantly obtained a 403 SignatureDoesNotMatch error when navigating to any of the Object Gateway sub menus. The moment we however then upgraded to Ceph Quincy 17.2.5 everything worked again perfectly.

The frustrating thing is that I couldn't figure out how to debug the RGW interactions initiated by Ceph MGR's dashboard.

Also really concerned that the problem with Pacific 16.2.11 is possibly due to a patch which will be included in the next version of Quincy, possibly 17.2.6.

NB: We validated this problem by upgrading another cluster where we could manage the Object Gateway via the Ceph dashboard which then also yielded the same problem after upgrading from 16.2.9 to 16.2.11, also remediated by simply then upgrading to 17.2.5.


I know Proxmox don't bundle/build cephadm,
  • Is it though expected that the dashboard doesn't connect to the defined RADOS gateway DNS FQDN as configured?
  • Anyone aware of patches having been merged relating to MGR's dashboard and its RGW calls?
  • Lastly, could anyone share tips on how to allow debugging the RGW interactions done by MGR's dashboard?


Ceph tracker:
https://tracker.ceph.com/issues/58811


Summary:
Attempted to enable debugging with the following:
Code: 
ceph dashboard debug enable
ceph tell mgr config set debug_mgr 20


Cepn dashboard / RGW configuration, as well as validation that S3 access and secret keys matched the system account 'dashboard':
Code: 
[admin@kvm7a ~]# radosgw-admin user info --uid=dashboard | grep _key.:
"access_key": "********************",
"secret_key": "****************************************"
[admin@kvm7a ~]# ceph config dump | grep -i -e rgw -e dash
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY ************
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY ********************************
mgr advanced mgr/dashboard/RGW_API_SSL_VERIFY false
mgr advanced mgr/dashboard/debug false
mgr advanced mgr/dashboard/server_addr 0.0.0.0
mgr advanced mgr/dashboard/ssl true
client.radosgw.kvm7a advanced rgw_dns_name kvm7a.redacted.com
client.radosgw.kvm7a basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem
client.radosgw.kvm7b advanced rgw_dns_name kvm7b.redacted.com
client.radosgw.kvm7b basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem
client.radosgw.kvm7e advanced rgw_dns_name kvm7e.redacted.com
client.radosgw.kvm7e basic rgw_frontends beast ssl_port=7480 ssl_certificate=/etc/pve/local/radosgw-ssl.pem


Herewith is the section from the mgr log after enabling debugging with a debug level of 20. It enumerates variables but then doesn't actually log the RGW interactions which are failing:
Code: 
2023-02-19T12:48:04.965+0200 7f24607fb700 10 mgr.server ms_handle_authentication ms_handle_authentication new session 0x5592faf37200 con 0x55930353e400 entity client.admin addr
2023-02-19T12:48:04.965+0200 7f24607fb700 10 mgr.server ms_handle_authentication session 0x5592faf37200 client.admin has caps allow * 'allow *'
2023-02-19T12:48:05.001+0200 7f24489c1700 10 module pg_autoscaler health checks:



2023-02-19T12:48:05.001+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/mirror_snapshot_schedule
2023-02-19T12:48:05.001+0200 7f2443b78700 10 mgr get_typed_config mirror_snapshot_schedule not found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/AUDIT_API_ENABLED
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config AUDIT_API_ENABLED not found
2023-02-19T12:48:05.025+0200 7f24275c0700 4 mgr get_store get_store key: mgr/dashboard/jwt_token_block_list
2023-02-19T12:48:05.025+0200 7f24275c0700 10 ceph_store_get jwt_token_block_list found: {"cd1a1a80-bca1-48ba-89fe-cd4d061a043b": 1676602596}
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.025+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.025+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.029+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/trash_purge_schedule
2023-02-19T12:48:05.029+0200 7f2442375700 10 mgr get_typed_config trash_purge_schedule not found
2023-02-19T12:48:05.029+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/kvm7b/mirror_snapshot_schedule
2023-02-19T12:48:05.029+0200 7f2443b78700 20 mgr get_config key: mgr/rbd_support/mirror_snapshot_schedule
2023-02-19T12:48:05.029+0200 7f2443b78700 10 mgr get_typed_config [kvm7b/]mirror_snapshot_schedule not found
2023-02-19T12:48:05.029+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.049+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/kvm7b/trash_purge_schedule
2023-02-19T12:48:05.049+0200 7f2442375700 20 mgr get_config key: mgr/rbd_support/trash_purge_schedule
2023-02-19T12:48:05.049+0200 7f2442375700 10 mgr get_typed_config [kvm7b/]trash_purge_schedule not found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ADMIN_RESOURCE found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SSL_VERIFY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ACCESS_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ACCESS_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SECRET_KEY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SECRET_KEY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_SSL_VERIFY
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_SSL_VERIFY found
2023-02-19T12:48:05.089+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/RGW_API_ADMIN_RESOURCE
2023-02-19T12:48:05.089+0200 7f24275c0700 10 mgr get_typed_config get_typed_config RGW_API_ADMIN_RESOURCE found
2023-02-19T12:48:05.093+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2023-02-19T12:48:05.093+0200 7f24275c0700 10 mgr get_typed_config REST_REQUESTS_TIMEOUT not found
2023-02-19T12:48:05.093+0200 7f24275c0700 20 mgr get_config key: mgr/dashboard/REST_REQUESTS_TIMEOUT
2023-02-19T12:48:05.093+0200 7f24275c0700 10 mgr get_typed_config REST_REQUESTS_TIMEOUT not found
2023-02-19T12:48:05.181+0200 7f244c288700 10 mgr.server tick

Additional information:
Code: 
[admin@kvm7a ~]# ceph dashboard feature status
Feature 'rbd': enabled
Feature 'mirroring': enabled
Feature 'iscsi': enabled
Feature 'cephfs': enabled
Feature 'rgw': enabled
Feature 'nfs': enabled

[admin@kvm7a ~]# ceph mgr services | jq .dashboard
"https://10.250.1.3:8443/"


Regards
David Herselman
 
Last edited:
+1, still analysing...

dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","RequestId":"tx0000065b5ff6b88518825-0063fcc'
Click to expand...
Ps:
We had the same issue with previous versions, the only fix was to update to the next version.

Edit:

Update to v17/Quincy solved the problem again! :( :)
 
Last edited:
I've found out that the radosgw service was not running on any nodes ... solving this resolved the issue.
 
I'm having the same behavior, dashboard works except the object storage - and it worked for a few months and now I can't get it to work again.
Error on dashboard:

The Object Gateway Service is not configured​

Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx000006cd7fcc67cca5cc4-006433f' b'cde-95d0414-default","HostId":"95d0414-default-default"}')​


ceph-mgr log:

[dashboard INFO rgw_client] Found RGW daemon with configuration: host=pve298, port=7480, ssl=False
[dashboard ERROR rest_client] RGW REST API failed GET req status: 403
[dashboard ERROR rgw_client] RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","RequestId":"tx00000df88ed952f2ba37e-0064340'
b'121-95d0414-default","HostId":"95d0414-default-default"}')
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 417, in __init__
self.userid = self._get_user_id(self.admin_path) if self.got_keys_from_config \
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 530, in func_wrapper
return func(
File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 452, in _get_user_id
response = request()
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 323, in __call__
resp = self.rest_client.do_request(method, self._gen_path(), params,
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 447, in do_request
raise RequestException(
dashboard.rest_client.RequestException: RGW REST API failed request with status code 403
(b'{"Code":"SignatureDoesNotMatch","RequestId":"tx00000df88ed952f2ba37e-0064340'
b'121-95d0414-default","HostId":"95d0414-default-default"}')


user "dashboard" configured to access radosgw:

root@pve298:/etc/pve/priv# radosgw-admin user info --uid=dashboard | grep _key.: "access_key": "I1UBIIXIW6J1SMPXXXX", "secret_key": "oqkZmn3XqYDb2aor1B3LbDrLFxHUIFSb9mXXXX"

root@pve298:/etc/pve/priv# ceph config dump | grep -i -e rgw -e dash mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY I1UBIIXIW6J1SMP4XXXX * mgr advanced mgr/dashboard/RGW_API_ADMIN_RESOURCE dashboard * mgr advanced mgr/dashboard/RGW_API_SECRET_KEY oqkZmn3XqYDb2aor1B3LbDrLFxHUIFSb9mXXXX * mgr advanced mgr/dashboard/RGW_API_SSL_VERIFY false * mgr advanced mgr/dashboard/debug false * mgr advanced mgr/dashboard/server_addr X.X.2.98 * mgr advanced mgr/dashboard/server_port 8443 * mgr advanced mgr/dashboard/ssl false * ... client.radosgw.pve298 advanced rgw_dns_name X.X.31.98 * client.radosgw.pve298 advanced rgw_host pve298 *

X.X.2.98 and X.X.31.98 are on the same machine.

On other clusters where I had this issue I had to reset like here: https://github.com/rook/rook/issues/3026 and it worked.
Here, no matter what I do, I still have the same issue.

Object storage works as expected, my issue is with the dashboard only :(
 
Last edited:
  • Like
Reactions: David Herselman
Hi guys.

The issue is happening because the MGR is trying to connect to RGW with the hostname of RGW, instead the IP address.
To avoid this problem, comment the variable rgw_dns_name in ceph.conf and configure your zonegroup to use hostnames with the FQDN of your domain and all name of yours RGW, like this:

{
"api_name" : "regionname",
"default_placement" : "default-placement",
"endpoints" : [
],
"hostnames" : [
"yourname.domain.com",
"rgw1-name",
"rgw2-name"
],

https://docs.ceph.com/en/quincy/radosgw/s3/commons/
https://docs.ceph.com/en/quincy/radosgw/multisite/
https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c#rgw-zonegroup-configuration
 
  • Like
Reactions: woloss
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back