API timing out??? Cannot view CEPH storage

[vanyell]

Running a three node cluster. Just upgraded to version 8. Cannot check storage from webui.
From pveproxy logs:

::ffff:192.168.200.202 - root@pam [19/08/2023:22:37:43 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 774
::ffff:192.168.200.202 - root@pam [19/08/2023:22:37:49 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 767
::ffff:192.168.200.202 - root@pam [19/08/2023:22:37:54 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 744
::ffff:192.168.200.202 - root@pam [19/08/2023:22:38:26 +0800] "GET /api2/json/nodes/pve3/storage/VM-DATA/content?content=images HTTP/1.1" 500 -
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:44 +0800] "GET /api2/extjs/nodes/pve3/apt/repositories?_dc=1692456042679 HTTP/1.1" 200 1314
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:45 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 771
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:46 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 756
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:47 +0800] "GET /api2/json/nodes/pve3/storage/VM-DATA/content?content=rootdir HTTP/1.1" 500 -
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:47 +0800] "GET /api2/extjs/nodes/pve3/subscription?_dc=1692456042680 HTTP/1.1" 200 180
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:48 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 758
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:49 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 768
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:50 +0800] "GET /api2/json/nodes/pve3/status HTTP/1.1" 200 758
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:50 +0800] "GET /api2/json/nodes/pve3/storage/local/status HTTP/1.1" 596 -
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:50 +0800] "GET /api2/json/nodes/pve3/storage/local/rrddata?timeframe=hour&cf=AVERAGE HTTP/1.1" 596 -
::ffff:192.168.200.202 - root@pam [19/08/2023:22:40:52 +0800] "GET /api2/json/nodes/pve3/storage/local-lvm/content?content=rootdir HTTP/1.1" 500 -
::ffff:192.168.200.202 - root@pam [19/08/2023:22:41:00 +0800] "GET /api2/json/nodes/pve3/storage/ISO_Images/status HTTP/1.1" 500 -

Anything else that I can check for?

pveversion -v:

proxmox-ve: 8.0.2 (running kernel: 6.2.16-8-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
proxmox-kernel-helper: 8.0.3
pve-kernel-5.15: 7.4-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-5
proxmox-kernel-6.2.16-8-pve: 6.2.16-8
proxmox-kernel-6.2: 6.2.16-8
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph: 17.2.6-pve1+3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.7
libpve-guest-common-perl: 5.0.4
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libqb0: 1.0.5-1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
openvswitch-switch: 3.1.0-2
proxmox-backup-client: 3.0.2-1
proxmox-backup-file-restore: 3.0.2-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.3
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-4
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[vanyell]

I may have more info.

Have two CEPH pools: VM-DATA (created pre 8.0), NEW (created after upgrade to 8.0)

Running rbd -p {pool} ls -l , hangs on VM-DATA but runs fine with NEW pool

root@pve3:~# time rbd -p VM-DATA ls -l
^C
real 0m27.097s
user 0m0.243s
sys 0m0.080s
root@pve3:~# time rbd -p NEW ls -l
NAME SIZE PARENT FMT PROT LOCK
vm-103-disk-0 32 GiB 2

real 0m0.135s
user 0m0.074s
sys 0m0.056s

Any ideas on how to get the images on VM-DATA onto NEW pool under this scenario?