Hi,
i followed pve documentation on upgrading ceph pacific to quincy, so, full upgrade to last pve 7, reboot, change to quincy, upgrade, restart mons/mgrs/osds, all went ok. But after this i rebooted one upgraded host and osds doesn't come up:
How to get osds up? The /var/lib/ceph/osd/ceph-X directory is empty.
i followed pve documentation on upgrading ceph pacific to quincy, so, full upgrade to last pve 7, reboot, change to quincy, upgrade, restart mons/mgrs/osds, all went ok. But after this i rebooted one upgraded host and osds doesn't come up:
Code:
Jul 11 02:31:03 proxmox-03 systemd[1]: Starting Ceph object storage daemon osd.4...
Jul 11 02:31:03 proxmox-03 systemd[1]: Started Ceph object storage daemon osd.4.
Jul 11 02:31:03 proxmox-03 ceph-osd[9053]: 2023-07-11T02:31:03.229+0200 7faa969183c0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
Jul 11 02:31:03 proxmox-03 ceph-osd[9053]: 2023-07-11T02:31:03.229+0200 7faa969183c0 -1 AuthRegistry(0x55e57e5f8140) no keyring found at /var/lib/ceph/osd/ceph-4/keyring, disabling cephx
Jul 11 02:31:03 proxmox-03 ceph-osd[9053]: 2023-07-11T02:31:03.233+0200 7faa969183c0 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
Jul 11 02:31:03 proxmox-03 ceph-osd[9053]: 2023-07-11T02:31:03.233+0200 7faa969183c0 -1 AuthRegistry(0x7ffd31921e90) no keyring found at /var/lib/ceph/osd/ceph-4/keyring, disabling cephx
Jul 11 02:31:03 proxmox-03 ceph-osd[9053]: failed to fetch mon config (--no-mon-config to skip)
Jul 11 02:31:03 proxmox-03 systemd[1]: ceph-osd@4.service: Main process exited, code=exited, status=1/FAILURE
Jul 11 02:31:03 proxmox-03 systemd[1]: ceph-osd@4.service: Failed with result 'exit-code'.
Jul 11 02:31:13 proxmox-03 systemd[1]: ceph-osd@4.service: Scheduled restart job, restart counter is at 3.
Jul 11 02:31:13 proxmox-03 systemd[1]: Stopped Ceph object storage daemon osd.4.
Jul 11 02:31:13 proxmox-03 systemd[1]: ceph-osd@4.service: Start request repeated too quickly.
Jul 11 02:31:13 proxmox-03 systemd[1]: ceph-osd@4.service: Failed with result 'exit-code'.
Jul 11 02:31:13 proxmox-03 systemd[1]: Failed to start Ceph object storage daemon osd.4.
Code:
root@proxmox-03:/var/log# lvs
File descriptor 9 (pipe:[101154]) leaked on lvs invocation. Parent PID 15233: bash
File descriptor 11 (pipe:[101155]) leaked on lvs invocation. Parent PID 15233: bash
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-block-a72ae29a-e413-4848-bc70-c2c76563a567 ceph-220d4477-940f-4970-b3a7-1836b88b11fc -wi-a----- <1.75t
osd-block-2ff9ed5f-075a-4e42-bc22-da303006b98c ceph-4962b11b-3aaf-4a65-9c1f-c66cb9a4dbd8 -wi-a----- <1.75t
Code:
root@proxmox-03:/var/log# ceph -s
cluster:
id: 99c06983-733f-482b-b59e-ad21f55c20e3
health: HEALTH_WARN
Degraded data redundancy: 200020/600060 objects degraded (33.333%), 129 pgs degraded, 129 pgs undersized
services:
mon: 3 daemons, quorum proxmox-01,proxmox-02,proxmox-03 (age 42m)
mgr: proxmox-02(active, since 54m), standbys: proxmox-01, proxmox-03
osd: 6 osds: 4 up (since 55m), 4 in (since 21m)
data:
pools: 2 pools, 129 pgs
objects: 200.02k objects, 743 GiB
usage: 1.2 TiB used, 5.8 TiB / 7.0 TiB avail
pgs: 200020/600060 objects degraded (33.333%)
129 active+undersized+degradedHow to get osds up? The /var/lib/ceph/osd/ceph-X directory is empty.
