Recent content by Mikepop

This is a conflint with dashboard, to solve it just run ceph mgr module disable dashboard Regards

Nobody can suggest how to debug? I guess I'll have to try on ceph mail list... I'll post the solution to the issue if I can solve it.. Regards

Hi, I use a grafana multicluster long time ago and after upgrade I see an empty answer to the queries. Any idea how to debug this? root@int101:~# ceph config dump WHO MASK LEVEL OPTION VALUE RO mon advanced...

Hi @silvered.dragon Indeed, recreating failed osds solved my issues with osd's down on reboots. Regards

Thanks Aarom, I'm recreating them, but it takes about 2 or 3 hours each and cannot degrade service performance too much. It seems also nobody else it's in the same situation, so digging may not woth it. I'll let you know if it issue persists once osd's been updated. Regards

Here they are, same order: root@int101:~# ll /var/lib/ceph/osd/ceph-22/ total 56 -rw-r--r-- 1 root root 402 Mar 5 2018 activate.monmap -rw-r--r-- 1 ceph ceph 3 Mar 5 2018 active lrwxrwxrwx 1 root root 9 Aug 4 11:17 block -> /dev/sde2 -rw-r--r-- 1 ceph ceph 37 Mar 5 2018 block_uuid...

Yes, it does, the ones with the new layout are owned by root:disk but also old layout ones, thery have mixed permissions: root@int101:~# ls -l /dev/sde* brw-rw---- 1 root disk 8, 64 Aug 4 11:17 /dev/sde brw-rw---- 1 root disk 8, 65 Aug 4 11:17 /dev/sde1 brw-rw---- 1 ceph ceph 8, 66 Aug 4...

@aaron I have other osds, maybe the recreated ones only with one partition: root@int101:~# ls -l /dev/sda* brw-rw---- 1 root disk 8, 0 Aug 1 07:54 /dev/sda Regards

@aaron this is the output form osd 22 root@int101:~# ls -l /dev/sde* brw-rw---- 1 root disk 8, 64 Aug 1 07:54 /dev/sde brw-rw---- 1 root disk 8, 65 Aug 1 07:54 /dev/sde1 brw-rw---- 1 ceph ceph 8, 66 Aug 2 23:51 /dev/sde2 Regards

@aaron, I already did almost two weeks ago. I dedided to post it also here cause I thought it would be more visible. Regards

Hello, I have some problems after update, osds sometime came back down and out: 2021-08-01T07:55:38.718+0200 7f35a7603f00 -1 bluestore(/var/lib/ceph/osd/ceph-22/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-22/block: (13) Permission denied 2021-08-01T07:55:38.718+0200...

More info on this, sometimes I'm able to readd failing osd's a bit later without changing anything at all: 2021-08-01T07:55:38.718+0200 7f35a7603f00 -1 bluestore(/var/lib/ceph/osd/ceph-22/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-22/block: (13) Permission denied...

Also, have one or more osds down on node reboots: 2021-08-01T07:55:38.476+0200 7f22b9b69f00 -1 bluestore(/var/lib/ceph/osd/ceph-23/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-23/block: (13) Permission denied 2021-08-01T07:55:38.476+0200 7f22b9b69f00 1 bdev(0x55f022ef8400...

Finally, this was a FEX rebooting several times, nothing Proxmox's upgrade related. Regards

Some cluster info: Cluster info: # pveversion --verbose proxmox-ve: 7.0-2 (running kernel: 5.11.22-2-pve) pve-manager: 7.0-10 (running version: 7.0-10/d2f465d3) pve-kernel-5.11: 7.0-5 pve-kernel-helper: 7.0-5 pve-kernel-5.4: 6.4-4 pve-kernel-5.11.22-2-pve: 5.11.22-4...