[SOLVED] OSD's not starting after upgrade 18.2.4 -> 18.2.7

[FXKai]

Hi,

after running the latest apt dist-upgrade and rebooting one of our storage servers, all OSD's on this node refuse to start.
After downgrading all CEPH packages the OSD's start immediately.

Any help appreciated

Setup: 7 x CEPH Storage nodes, 3 x PVE Compute Nodes

Log files of the nodes:

2025-11-11T11:49:22.030+0100 7c8ce33e3940  0 set uid:gid to 64045:64045 (ceph:ceph)
2025-11-11T11:49:22.030+0100 7c8ce33e3940  0 ceph version 18.2.7 (4cac8341a72477c60a6f153f3ed344b49870c932) reef (stable), process ceph-osd, pid 8229
2025-11-11T11:49:22.030+0100 7c8ce33e3940  0 pidfile_write: ignore empty --pid-file
2025-11-11T11:49:22.032+0100 7c8ce33e3940  1 bdev(0x5dd69d9b1400 /var/lib/ceph/osd/ceph-81/block) open path /var/lib/ceph/osd/ceph-81/block
2025-11-11T11:49:22.032+0100 7c8ce33e3940  0 bdev(0x5dd69d9b1400 /var/lib/ceph/osd/ceph-81/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-81/block failed: (22) Invalid argument
2025-11-11T11:49:22.032+0100 7c8ce33e3940  1 bdev(0x5dd69d9b1400 /var/lib/ceph/osd/ceph-81/block) open size 3200627245056 (0x2e934400000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2025-11-11T11:49:22.033+0100 7c8ce33e3940 -1 bluestore(/var/lib/ceph/osd/ceph-81) _set_cache_sizes bluestore_cache_meta_ratio (0.8) + bluestore_cache_kv_ratio (0.2) + bluestore_cache_kv_onode_ratio (0.04) = 1.04; must be <= 1.0
2025-11-11T11:49:22.033+0100 7c8ce33e3940  1 bdev(0x5dd69d9b1400 /var/lib/ceph/osd/ceph-81/block) close
2025-11-11T11:49:22.306+0100 7c8ce33e3940  0 starting osd.81 osd_data /var/lib/ceph/osd/ceph-81 /var/lib/ceph/osd/ceph-81/journal
2025-11-11T11:49:22.322+0100 7c8ce33e3940  0 load: jerasure load: lrc
2025-11-11T11:49:22.322+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open path /var/lib/ceph/osd/ceph-81/block
2025-11-11T11:49:22.322+0100 7c8ce33e3940  0 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-81/block failed: (22) Invalid argument
2025-11-11T11:49:22.323+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open size 3200627245056 (0x2e934400000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2025-11-11T11:49:22.323+0100 7c8ce33e3940 -1 bluestore(/var/lib/ceph/osd/ceph-81) _set_cache_sizes bluestore_cache_meta_ratio (0.8) + bluestore_cache_kv_ratio (0.2) + bluestore_cache_kv_onode_ratio (0.04) = 1.04; must be <= 1.0
2025-11-11T11:49:22.323+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) close
2025-11-11T11:49:22.596+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open path /var/lib/ceph/osd/ceph-81/block
2025-11-11T11:49:22.596+0100 7c8ce33e3940  0 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-81/block failed: (22) Invalid argument
2025-11-11T11:49:22.597+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open size 3200627245056 (0x2e934400000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational device, discard supporte
d
2025-11-11T11:49:22.597+0100 7c8ce33e3940 -1 bluestore(/var/lib/ceph/osd/ceph-81) _set_cache_sizes bluestore_cache_meta_ratio (0.8) + bluestore_cache_kv_ratio (0.2) + bluestore_cache_kv_onode_ratio (0.04) = 1.0
4; must be <= 1.0
2025-11-11T11:49:22.597+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) close
2025-11-11T11:49:22.871+0100 7c8ce33e3940  1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_per_io: 499321.90 bytes/io, osd_bandwidth_capacity_per_shard 157286400.00 bytes/second
2025-11-11T11:49:22.871+0100 7c8ce33e3940  0 osd.81:0.OSDShard using op scheduler mclock_scheduler, cutoff=196
2025-11-11T11:49:22.873+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open path /var/lib/ceph/osd/ceph-81/block
2025-11-11T11:49:22.873+0100 7c8ce33e3940  0 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-81/block failed: (22) Invalid argument
2025-11-11T11:49:22.873+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) open size 3200627245056 (0x2e934400000, 2.9 TiB) block_size 4096 (4 KiB) non-rotational device, discard supporte
d
2025-11-11T11:49:22.874+0100 7c8ce33e3940 -1 bluestore(/var/lib/ceph/osd/ceph-81) _set_cache_sizes bluestore_cache_meta_ratio (0.8) + bluestore_cache_kv_ratio (0.2) + bluestore_cache_kv_onode_ratio (0.04) = 1;
must be <= 1.0
2025-11-11T11:49:22.874+0100 7c8ce33e3940  1 bdev(0x5dd69e761400 /var/lib/ceph/osd/ceph-81/block) close
2025-11-11T11:49:23.147+0100 7c8ce33e3940 -1 osd.81 0 OSD:init: unable to mount object store
2025-11-11T11:49:23.147+0100 7c8ce33e3940 -1 ESC[0;31m ** ERROR: osd init failed: (22) Invalid argumentESC[0m

 

[FXKai]

We found a kind of solution ourselves. We had caching values set in our ceph.conf. Apparently, this is what Ceph has become stricter about since version 18.2.7 or higher (we are now on 19.2.3). The startup checks have become more picky with this.

Example error message:

(/var/lib/ceph/osd/ceph-81) _set_cache_sizes bluestore_cache_meta_ratio (0.8) + bluestore_cache_kv_ratio (0.2) + bluestore_cache_kv_onode_ratio (0.04) = 1;
must be <= 1.0

We tried setting various values, first in ceph.conf:

[osd]
    bluestore_cache_autotune = false
    bluestore_cache_kv_ratio = 0.2
    bluestore_cache_kv_onode_ratio = 0
    bluestore_cache_meta_ratio = 0.76

Then globally via the CLI:

ceph config get osd bluestore_cache_meta_ratio
ceph config get osd bluestore_cache_kv_ratio
ceph config get osd bluestore_cache_kv_onode_ratio
ceph config get osd bluestore_cache_autotune

# then
ceph config set osd bluestore_cache_meta_ratio 0.76
ceph config set osd bluestore_cache_kv_ratio 0.2
ceph config set osd bluestore_cache_kv_onode_ratio 0.0
ceph config set osd bluestore_cache_autotune false

Restarts did not work initially, so we destroyed one OSD and recreated it. At first, the creation also failed, but as soon as the cache values globally met the <=1.0 criteria, it successfully created a new OSD.


Meanwhile, we stopped all OSDs on the node:

systemctl stop ceph-osd.target

After recreating one of the OSDs, we were also able to start the "old" OSDs. Most likely, they picked up the new cache strategy config now.


Note: We performed all of this on a single node while leaving all other nodes untouched to avoid downtime or data loss!