BlueFS spillover very dispair on disks

samumsq

New Member
Aug 27, 2025
14
1
3
Hello everyone,
We have a proxmox cluster and we are using Ceph 19.2.2. Our cluster has 3 nodes, in each node we have an SSD that we use for all OSDs db, and 5 HDD.
Yesterday, we started to experience BlueFS spillover on only 2 OSDs. After checking we see that this 2 OSD (OSD 5 and OSD 8) use almost all db space, but all of the others barely use 5GB
ceph df detail
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 33.71567 - 34 TiB 20 TiB 19 TiB 17 MiB 261 GiB 14 TiB 58.01 1.00 - root default
-3 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 63 GiB 4.7 TiB 57.97 1.00 - host pve01-poz
1 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 5.6 MiB 3.7 GiB 655 GiB 65.86 1.14 56 up osd.1
2 hdd 1.87309 1.00000 1.9 TiB 994 GiB 939 GiB 24 KiB 3.3 GiB 924 GiB 51.85 0.89 45 up osd.2
3 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 30 KiB 3.3 GiB 747 GiB 61.08 1.05 53 up osd.3
4 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 23 KiB 2.9 GiB 648 GiB 66.20 1.14 58 up osd.4
10 hdd 1.87309 1.00000 1.9 TiB 942 GiB 887 GiB 31 KiB 3.2 GiB 976 GiB 49.14 0.85 44 up osd.10
15 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 975 GiB 36 KiB 46 GiB 888 GiB 53.72 0.93 49 up osd.15
-5 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 175 GiB 4.7 TiB 58.09 1.00 - host pve02-poz
5 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 27 KiB 68 GiB 701 GiB 63.62 1.10 55 up osd.5
6 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 983 GiB 5.6 MiB 31 GiB 881 GiB 54.09 0.93 47 up osd.6
7 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 37 KiB 2.7 GiB 716 GiB 62.87 1.08 56 up osd.7
8 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 1013 GiB 23 KiB 57 GiB 848 GiB 55.79 0.96 48 up osd.8
9 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 46 KiB 2.9 GiB 776 GiB 59.54 1.03 52 up osd.9
16 hdd 1.87309 1.00000 1.9 TiB 1009 GiB 954 GiB 35 KiB 13 GiB 909 GiB 52.59 0.91 47 up osd.16
-7 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 23 GiB 4.7 TiB 57.97 1.00 - host pve03-poz
0 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 25 KiB 3.1 GiB 794 GiB 58.62 1.01 48 up osd.0
11 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 971 GiB 18 KiB 3.5 GiB 892 GiB 53.48 0.92 48 up osd.11
12 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 5.6 MiB 3.0 GiB 735 GiB 61.67 1.06 55 up osd.12
13 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 39 KiB 2.9 GiB 824 GiB 57.06 0.98 49 up osd.13
14 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 25 KiB 3.7 GiB 653 GiB 65.96 1.14 60 up osd.14
17 hdd 1.87309 1.00000 1.9 TiB 979 GiB 924 GiB 18 KiB 6.9 GiB 939 GiB 51.05 0.88 45 up osd.17

Is this normal behaviour for Ceph? Should we check anything before buying a bigger SSD for OSDs DB?
Thanks in advance for your help
 
Hi samumsq!

There's quite a high meta data consumption in general from the output. That CEPH cluster is running on very few disks, it's intended for hundreds of disks so strange behaviour can occur in smaller populations. Taking a look at the meta data distribution, though:
OSDMETA
568 GiB
857 GiB
1546 GiB
631 GiB
1613 GiB
most others~3 GiB

You can clearly see that some OSDs are producing more, which points to an FS oddity. Looking at it from a per host perspective, you have:
  • pve01: about 63 GiB META
  • pve02: about 175 GiB META
  • pve03: about 23 GiB META
Which is telling us that your second node has something going on with it. But don't take my word for it, the Squid Documentation outlined what normal DB usage looks like for your HDD OSDs (1%-3% size of disk), which at about 1.87 TiB each, the rough guidelines would say:
  • 1% ≈ 19 GiB
  • 2% ≈ 38 GiB
  • 3% ≈ 56 GiB
  • 4% ≈ 77 GiB

It's too opaque / lacking in detail for me to say what is wrong but I can see that something looks wrong between that list and the list above for pve02. Can you please run the following commands?
Code:
ceph health detail
ceph daemon osd.5 bluestore bluefs device info
ceph daemon osd.7 bluestore bluefs device info
ceph daemon osd.8 bluestore bluefs device info
ceph daemon osd.15 bluestore bluefs device info
ceph-volume lvm list
lvs -a -o vg_name,lv_name,lv_size,devices

bluefs device info output is useful here as it tells you how much space BlueFS actually has on BDEV_DB and the slow device rather than trying to infer DB occupancy from ceph df detail.

That's as much as I can help for now, gotta get back to my day job and finish another draft question!
Thanks,


Tmanok
 
Last edited:
Hi Tmanok,

First of all thanks for your help.
Right now there's no spillage, i hope the commands are still usefull to troubleshoot the cause.
Here you have the output of the commands, if there's something else you need please let me know

ceph health detail

root@pve02-poz:~# ceph health detail

ceph daemon osd. bluestore device info (i think there's something i'm missing with this command because it shows a message saying Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n")
root@pve02-poz:~# ceph daemon osd.5 bluestore bluefs device info
ceph daemon osd.7 bluestore bluefs device info
ceph daemon osd.8 bluestore bluefs device info
ceph daemon osd.15 bluestore bluefs device info
{
"dev": {
"device": "BDEV_DB",
"total": 75161919488,
"free": 64615342080,
"bluefs_used": 10546577408
},
"dev": {
"device": "BDEV_SLOW",
"total": 2000397795328,
"free": 754588286976,
"bluefs_used": 0,
"bluefs max available": 675616587776
}
}
{
"dev": {
"device": "BDEV_DB",
"total": 69793210368,
"free": 66760728576,
"bluefs_used": 3032481792
},
"dev": {
"device": "BDEV_SLOW",
"total": 2000397795328,
"free": 766239342592,
"bluefs_used": 0,
"bluefs max available": 682897571840
}
}
{
"dev": {
"device": "BDEV_DB",
"total": 64424501248,
"free": 34816909312,
"bluefs_used": 29607591936
},
"dev": {
"device": "BDEV_SLOW",
"total": 2000397795328,
"free": 910491742208,
"bluefs_used": 0,
"bluefs max available": 830632820736
}
}
Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n"

ceph-volume lvm list
root@pve02-poz:~# ceph-volume lvm list


====== osd.16 ======

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-5ece6002-5355-468c-80e4-d95f45075377

block device /dev/ceph-b4eb6470-a972-4d7a-8302-347d6a5ff614/osd-block-b195cd17-5dda-4532-8433-128f89cf3648
block uuid AiOIeR-Wn9c-oSew-GBRF-9jYJ-QW8B-rTcJ1b
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-5ece6002-5355-468c-80e4-d95f45075377
db uuid wN0ZOf-Mvx1-Zr4Y-M4C3-EBH7-ofng-U1QdPQ
encrypted 0
osd fsid b195cd17-5dda-4532-8433-128f89cf3648
osd id 16
osdspec affinity
type db
vdo 0
devices /dev/sdc

[block] /dev/ceph-b4eb6470-a972-4d7a-8302-347d6a5ff614/osd-block-b195cd17-5dda-4532-8433-128f89cf3648

block device /dev/ceph-b4eb6470-a972-4d7a-8302-347d6a5ff614/osd-block-b195cd17-5dda-4532-8433-128f89cf3648
block uuid AiOIeR-Wn9c-oSew-GBRF-9jYJ-QW8B-rTcJ1b
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-5ece6002-5355-468c-80e4-d95f45075377
db uuid wN0ZOf-Mvx1-Zr4Y-M4C3-EBH7-ofng-U1QdPQ
encrypted 0
osd fsid b195cd17-5dda-4532-8433-128f89cf3648
osd id 16
osdspec affinity
type block
vdo 0
devices /dev/sdg

====== osd.5 =======

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-336c9a2d-f9d0-492b-9070-aff68b8c0a30

block device /dev/ceph-d8af176f-db0a-4636-a978-44a60c7a220d/osd-block-ed36875b-15a2-47f0-bc24-4ba478205f28
block uuid pt5p0d-2gWJ-dSOB-EW58-nYyG-dimb-JrBPB9
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-336c9a2d-f9d0-492b-9070-aff68b8c0a30
db uuid tCoNqn-qLnM-2VRh-SWS3-AU7E-EIG4-Rp3ubM
encrypted 0
osd fsid ed36875b-15a2-47f0-bc24-4ba478205f28
osd id 5
osdspec affinity
type db
vdo 0
devices /dev/sdc,/dev/sdc

[block] /dev/ceph-d8af176f-db0a-4636-a978-44a60c7a220d/osd-block-ed36875b-15a2-47f0-bc24-4ba478205f28

block device /dev/ceph-d8af176f-db0a-4636-a978-44a60c7a220d/osd-block-ed36875b-15a2-47f0-bc24-4ba478205f28
block uuid pt5p0d-2gWJ-dSOB-EW58-nYyG-dimb-JrBPB9
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-336c9a2d-f9d0-492b-9070-aff68b8c0a30
db uuid tCoNqn-qLnM-2VRh-SWS3-AU7E-EIG4-Rp3ubM
encrypted 0
osd fsid ed36875b-15a2-47f0-bc24-4ba478205f28
osd id 5
osdspec affinity
type block
vdo 0
devices /dev/sda

====== osd.6 =======

[block] /dev/ceph-67e00c22-7884-4124-af9f-57090344d6c4/osd-block-8f266de2-52b1-4555-8862-b9c831f6f651

block device /dev/ceph-67e00c22-7884-4124-af9f-57090344d6c4/osd-block-8f266de2-52b1-4555-8862-b9c831f6f651
block uuid RdOLxn-MfdV-trRi-UFgm-oB8Y-AJ0s-fKWLqF
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-04f782d2-e7ed-48a6-b3c7-1f74b688823e
db uuid 7GII0G-AvQe-5z6f-Wehg-l2MJ-HHTW-uM3r0f
encrypted 0
osd fsid 8f266de2-52b1-4555-8862-b9c831f6f651
osd id 6
osdspec affinity
type block
vdo 0
devices /dev/sdb

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-04f782d2-e7ed-48a6-b3c7-1f74b688823e

block device /dev/ceph-67e00c22-7884-4124-af9f-57090344d6c4/osd-block-8f266de2-52b1-4555-8862-b9c831f6f651
block uuid RdOLxn-MfdV-trRi-UFgm-oB8Y-AJ0s-fKWLqF
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-04f782d2-e7ed-48a6-b3c7-1f74b688823e
db uuid 7GII0G-AvQe-5z6f-Wehg-l2MJ-HHTW-uM3r0f
encrypted 0
osd fsid 8f266de2-52b1-4555-8862-b9c831f6f651
osd id 6
osdspec affinity
type db
vdo 0
devices /dev/sdc

====== osd.7 =======

[block] /dev/ceph-673444e8-e530-4ea9-8018-211516ba479c/osd-block-f7fe52ed-e421-46f5-8b24-b7c61e86475b

block device /dev/ceph-673444e8-e530-4ea9-8018-211516ba479c/osd-block-f7fe52ed-e421-46f5-8b24-b7c61e86475b
block uuid YgljRg-TzGU-xZQo-aeTh-myFX-qhRD-u31Z3z
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c3bdd003-a085-48eb-a255-6af4f285baf0
db uuid vsXXmV-pBd2-9PAd-Ep1o-jkso-8fOP-NajlSm
encrypted 0
osd fsid f7fe52ed-e421-46f5-8b24-b7c61e86475b
osd id 7
osdspec affinity
type block
vdo 0
devices /dev/sdd

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c3bdd003-a085-48eb-a255-6af4f285baf0

block device /dev/ceph-673444e8-e530-4ea9-8018-211516ba479c/osd-block-f7fe52ed-e421-46f5-8b24-b7c61e86475b
block uuid YgljRg-TzGU-xZQo-aeTh-myFX-qhRD-u31Z3z
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c3bdd003-a085-48eb-a255-6af4f285baf0
db uuid vsXXmV-pBd2-9PAd-Ep1o-jkso-8fOP-NajlSm
encrypted 0
osd fsid f7fe52ed-e421-46f5-8b24-b7c61e86475b
osd id 7
osdspec affinity
type db
vdo 0
devices /dev/sdc,/dev/sdc

====== osd.8 =======

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-56453f4f-89da-4ad2-a8a6-ef7c955ca5e3

block device /dev/ceph-f716745e-4e5e-4d54-b85d-d4d086e9f1ef/osd-block-fa82c023-7150-439f-a157-7c58cc0a860f
block uuid ur66xt-TOUx-4ua6-VhKr-ibHA-bTEX-5izP2y
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-56453f4f-89da-4ad2-a8a6-ef7c955ca5e3
db uuid sf1Q5q-cgwA-SC8n-0vSy-Gevn-mCSB-1ZPunL
encrypted 0
osd fsid fa82c023-7150-439f-a157-7c58cc0a860f
osd id 8
osdspec affinity
type db
vdo 0
devices /dev/sdc,/dev/sdc

[block] /dev/ceph-f716745e-4e5e-4d54-b85d-d4d086e9f1ef/osd-block-fa82c023-7150-439f-a157-7c58cc0a860f

block device /dev/ceph-f716745e-4e5e-4d54-b85d-d4d086e9f1ef/osd-block-fa82c023-7150-439f-a157-7c58cc0a860f
block uuid ur66xt-TOUx-4ua6-VhKr-ibHA-bTEX-5izP2y
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-56453f4f-89da-4ad2-a8a6-ef7c955ca5e3
db uuid sf1Q5q-cgwA-SC8n-0vSy-Gevn-mCSB-1ZPunL
encrypted 0
osd fsid fa82c023-7150-439f-a157-7c58cc0a860f
osd id 8
osdspec affinity
type block
vdo 0
devices /dev/sde

====== osd.9 =======

[block] /dev/ceph-551379bc-0cf4-457f-82d7-e1dd087674d5/osd-block-94ad62ba-1942-411f-8102-73e7d4d0a0c3

block device /dev/ceph-551379bc-0cf4-457f-82d7-e1dd087674d5/osd-block-94ad62ba-1942-411f-8102-73e7d4d0a0c3
block uuid N6UJIg-ZLLO-cKzm-D45F-JEtZ-oY0k-EKSPV7
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c8ab2f8b-bb08-4212-8d1b-8adb2f6cc24a
db uuid xLEjMa-XI5e-EFee-x0tb-CL3n-lCW2-brug52
encrypted 0
osd fsid 94ad62ba-1942-411f-8102-73e7d4d0a0c3
osd id 9
osdspec affinity
type block
vdo 0
devices /dev/sdf

[db] /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c8ab2f8b-bb08-4212-8d1b-8adb2f6cc24a

block device /dev/ceph-551379bc-0cf4-457f-82d7-e1dd087674d5/osd-block-94ad62ba-1942-411f-8102-73e7d4d0a0c3
block uuid N6UJIg-ZLLO-cKzm-D45F-JEtZ-oY0k-EKSPV7
cephx lockbox secret
cluster fsid acb36c90-6f0c-462b-8857-7c277ee2b85a
cluster name ceph
crush device class
db device /dev/ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940/osd-db-c8ab2f8b-bb08-4212-8d1b-8adb2f6cc24a
db uuid xLEjMa-XI5e-EFee-x0tb-CL3n-lCW2-brug52
encrypted 0
osd fsid 94ad62ba-1942-411f-8102-73e7d4d0a0c3
osd id 9
osdspec affinity
type db
vdo 0
devices /dev/sdc

lvs -a -o vg_name,lv_name,lv_size,devices
root@pve02-poz:~# lvs -a -o vg_name,lv_name,lv_size,devices
VG LV LSize Devices
ceph-551379bc-0cf4-457f-82d7-e1dd087674d5 osd-block-94ad62ba-1942-411f-8102-73e7d4d0a0c3 <1,82t /dev/sdf(0)
ceph-673444e8-e530-4ea9-8018-211516ba479c osd-block-f7fe52ed-e421-46f5-8b24-b7c61e86475b <1,82t /dev/sdd(0)
ceph-67e00c22-7884-4124-af9f-57090344d6c4 osd-block-8f266de2-52b1-4555-8862-b9c831f6f651 <1,82t /dev/sdb(0)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-04f782d2-e7ed-48a6-b3c7-1f74b688823e 55,00g /dev/sdc(14080)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-336c9a2d-f9d0-492b-9070-aff68b8c0a30 70,00g /dev/sdc(0)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-336c9a2d-f9d0-492b-9070-aff68b8c0a30 70,00g /dev/sdc(87040)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-56453f4f-89da-4ad2-a8a6-ef7c955ca5e3 60,00g /dev/sdc(42240)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-56453f4f-89da-4ad2-a8a6-ef7c955ca5e3 60,00g /dev/sdc(90880)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-5ece6002-5355-468c-80e4-d95f45075377 55,00g /dev/sdc(70400)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-c3bdd003-a085-48eb-a255-6af4f285baf0 65,00g /dev/sdc(28160)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-c3bdd003-a085-48eb-a255-6af4f285baf0 65,00g /dev/sdc(84480)
ceph-985eb0a8-79b6-4065-b2cf-44c2fa289940 osd-db-c8ab2f8b-bb08-4212-8d1b-8adb2f6cc24a 55,00g /dev/sdc(56320)
ceph-b4eb6470-a972-4d7a-8302-347d6a5ff614 osd-block-b195cd17-5dda-4532-8433-128f89cf3648 <1,82t /dev/sdg(0)
ceph-d8af176f-db0a-4636-a978-44a60c7a220d osd-block-ed36875b-15a2-47f0-bc24-4ba478205f28 <1,82t /dev/sda(0)
ceph-f716745e-4e5e-4d54-b85d-d4d086e9f1ef osd-block-fa82c023-7150-439f-a157-7c58cc0a860f <1,82t /dev/sde(0)
pve root <55,88g /dev/sdh5(3814)
pve swap <14,90g /dev/sdh5(0)

Thanks again for your help
 
Hey samumsq,

Woah that's different. What happened since your first post? Did you do anything to the CEPH cluster in the meantime?? This is surprising, and if you haven't done anything then my best guess is you have something transient here. Your OSDs are nowhere near full anymore, so either way something has changed.

Don't worry about that command failing, by the way, OSD 15 actually belongs to pve01, so running ceph daemon osd.15 on pve02 won't find its local administration socket (shoulda specified, sorry). You can execute it on pve01 or use: ceph tell osd.15 bluestore bluefs device info

Reviewing it more carefully, I think I made an error in my earlier comment, it feels like this could be due to RocksDB, whic periodically rewrites and consolidates its SST files. During a sufficiently large compaction, temporary files can coexist with the files they will replace, causing disk usage to rise significantly before falling again.

Here's a command dump that I think we should try, because unfortunately the output you gave can only tell us what happened so far into the past:
Code:
ceph tell osd.5 perf dump -f json | jq '{bluefs, rocksdb}'
ceph tell osd.8 perf dump -f json | jq '{bluefs, rocksdb}'
journalctl -u ceph-osd@5 --since "2026-10-12" | grep -Ei 'bluefs|spillover|rocksdb|compact'
journalctl -u ceph-osd@8 --since "2026-10-12" | grep -Ei 'bluefs|spillover|rocksdb|compact'

Cheers,


Tmanok
 
Hi Tmanok,

Thanks again for your reply.
My apologies because yerterday i had no spilage because a colleage give a bit more space to Metadata in each disk to solve temporarily the problem.
There's still a huge difference between disks
ceph osd df tree
ceph df detail
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 33.71567 - 34 TiB 20 TiB 19 TiB 17 MiB 182 GiB 14 TiB 58.15 1.00 - root default
-3 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 63 GiB 4.7 TiB 58.12 1.00 - host pve01-poz
1 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 5.6 MiB 3.0 GiB 652 GiB 66.01 1.14 56 up osd.1
2 hdd 1.87309 1.00000 1.9 TiB 997 GiB 942 GiB 27 KiB 3.7 GiB 921 GiB 51.98 0.89 45 up osd.2
3 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 24 KiB 3.4 GiB 744 GiB 61.23 1.05 53 up osd.3
4 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 28 KiB 2.9 GiB 645 GiB 66.35 1.14 58 up osd.4
10 hdd 1.87309 1.00000 1.9 TiB 945 GiB 890 GiB 36 KiB 3.2 GiB 973 GiB 49.26 0.85 44 up osd.10
15 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 978 GiB 41 KiB 47 GiB 885 GiB 53.87 0.93 49 up osd.15
-5 11.23856 - 11 TiB 6.6 TiB 6.2 TiB 5.7 MiB 95 GiB 4.7 TiB 58.22 1.00 - host pve02-poz
5 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 12 KiB 12 GiB 702 GiB 63.67 1.09 55 up osd.5
6 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 985 GiB 5.6 MiB 33 GiB 878 GiB 54.23 0.93 47 up osd.6
7 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 42 KiB 2.7 GiB 713 GiB 63.03 1.08 56 up osd.7
8 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1016 GiB 20 KiB 30 GiB 847 GiB 55.94 0.96 48 up osd.8
9 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 38 KiB 2.5 GiB 773 GiB 59.70 1.03 52 up osd.9
16 hdd 1.87309 1.00000 1.9 TiB 1011 GiB 956 GiB 40 KiB 15 GiB 907 GiB 52.73 0.91 47 up osd.16
-7 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 24 GiB 4.7 TiB 58.12 1.00 - host pve03-poz
0 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 30 KiB 2.6 GiB 791 GiB 58.76 1.01 48 up osd.0
11 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 973 GiB 23 KiB 3.1 GiB 890 GiB 53.62 0.92 48 up osd.11
12 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 5.6 MiB 3.2 GiB 732 GiB 61.82 1.06 55 up osd.12
13 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 42 KiB 4.1 GiB 821 GiB 57.19 0.98 49 up osd.13
14 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 32 KiB 2.3 GiB 650 GiB 66.12 1.14 60 up osd.14
17 hdd 1.87309 1.00000 1.9 TiB 982 GiB 927 GiB 24 KiB 9.1 GiB 936 GiB 51.18 0.88 45 up osd.17
TOTAL 34 TiB 20 TiB 19 TiB 17 MiB 182 GiB 14 TiB 58.15
MIN/MAX VAR: 0.85/1.14 STDDEV: 5.41
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 34 TiB 14 TiB 20 TiB 20 TiB 58.15
TOTAL 34 TiB 14 TiB 20 TiB 20 TiB 58.15

--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
.mgr 3 1 66 MiB 60 MiB 5.5 MiB 20 197 MiB 181 MiB 17 MiB 0 3.2 TiB N/A N/A N/A 0 B 0 B
Storage_Vms 4 256 5.3 TiB 5.3 TiB 108 KiB 1.78M 16 TiB 16 TiB 324 KiB 62.13 3.2 TiB N/A N/A N/A 0 B 0 B
Storage_CephS_data 5 32 948 GiB 948 GiB 0 B 242.82k 2.8 TiB 2.8 TiB 0 B 22.34 3.2 TiB N/A N/A N/A 0 B 0 B
Storage_CephS_metadata 6 16 172 MiB 172 MiB 30 KiB 70 516 MiB 516 MiB 90 KiB 0 3.2 TiB N/A N/A N/A 0 B 0 B

If there's anything else you need to know please let us know
Thanks for your time

Alse here's the output of the command you told me to execute
ceph tell osd.5 perf dump -f json | jq '{bluefs, rocksdb}'
ceph tell osd.8 perf dump -f json | jq '{bluefs, rocksdb}'
journalctl -u ceph-osd@5 --since "2026-10-12" | grep -Ei 'bluefs|spillover|rocksdb|compact'
journalctl -u ceph-osd@8 --since "2026-10-12" | grep -Ei 'bluefs|spillover|rocksdb|compact'
{
"bluefs": {
"db_total_bytes": 75161919488,
"db_used_bytes": 13092519936,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 2000397795328,
"slow_used_bytes": 0,
"num_files": 253,
"log_bytes": 14913536,
"log_compactions": 2252,
"log_write_count": 8069720,
"logged_bytes": 33053573120,
"files_written_wal": 8739,
"files_written_sst": 1314,
"write_count_wal": 8055766,
"write_count_sst": 46818,
"bytes_written_wal": 176855584768,
"bytes_written_sst": 24571727872,
"bytes_written_slow": 0,
"max_bytes_wal": 0,
"max_bytes_db": 51437895680,
"max_bytes_slow": 0,
"alloc_unit_slow": 65536,
"alloc_unit_db": 1048576,
"alloc_unit_wal": 0,
"read_random_lat": {
"avgcount": 771481,
"sum": 16.723545462,
"avgtime": 2.1677e-05
},
"read_random_count": 771481,
"read_random_bytes": 26169882981,
"read_random_disk_count": 248946,
"read_random_disk_bytes": 24688731604,
"read_random_disk_bytes_wal": 0,
"read_random_disk_bytes_db": 24550952043,
"read_random_disk_bytes_slow": 137779561,
"read_random_buffer_count": 524197,
"read_random_buffer_bytes": 1481151377,
"read_lat": {
"avgcount": 125360,
"sum": 5.359086364,
"avgtime": 4.2749e-05
},
"read_count": 125360,
"read_bytes": 3977205447,
"read_disk_count": 93942,
"read_disk_bytes": 3064020992,
"read_disk_bytes_wal": 0,
"read_disk_bytes_db": 3062026240,
"read_disk_bytes_slow": 1998848,
"read_prefetch_count": 125263,
"read_prefetch_bytes": 3974482470,
"write_count": 16175577,
"write_disk_count": 16177912,
"write_bytes": 239236014080,
"compact_lat": {
"avgcount": 2252,
"sum": 23.429906385,
"avgtime": 0.010404043
},
"compact_lock_lat": {
"avgcount": 2252,
"sum": 1.350269098,
"avgtime": 0.000599586
},
"fsync_lat": {
"avgcount": 8067000,
"sum": 3628.817720852,
"avgtime": 0.000449834
},
"flush_lat": {
"avgcount": 8103605,
"sum": 2142.169290794,
"avgtime": 0.000264347
},
"unlink_lat": {
"avgcount": 10555,
"sum": 0.12053331,
"avgtime": 1.1419e-05
},
"truncate_lat": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"alloc_slow_fallback": 0,
"alloc_slow_size_fallback": 0,
"read_zeros_candidate": 0,
"read_zeros_errors": 0,
"wal_alloc_lat": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"db_alloc_lat": {
"avgcount": 27510,
"sum": 0.330347497,
"avgtime": 1.2008e-05
},
"slow_alloc_lat": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"alloc_wal_max_lat": 0,
"alloc_db_max_lat": 0.003345333,
"alloc_slow_max_lat": 0
},
"rocksdb": {
"get_latency": {
"avgcount": 1661166,
"sum": 53.633803402,
"avgtime": 3.2286e-05
},
"submit_latency": {
"avgcount": 10010390,
"sum": 813.516372663,
"avgtime": 8.1267e-05
},
"submit_sync_latency": {
"avgcount": 8045894,
"sum": 3731.23752681,
"avgtime": 0.000463744
},
"compact": 0,
"compact_running": 0,
"compact_completed": 0,
"compact_lasted": 0,
"compact_queue_merge": 0,
"compact_queue_len": 0,
"rocksdb_write_wal_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_memtable_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_delay_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_pre_and_post_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
}
}
}
{
"bluefs": {
"db_total_bytes": 64424501248,
"db_used_bytes": 31445745664,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 2000397795328,
"slow_used_bytes": 0,
"num_files": 471,
"log_bytes": 13541376,
"log_compactions": 2029,
"log_write_count": 7270668,
"logged_bytes": 29780656128,
"files_written_wal": 8798,
"files_written_sst": 1366,
"write_count_wal": 7258082,
"write_count_sst": 47800,
"bytes_written_wal": 174856781824,
"bytes_written_sst": 25043046400,
"bytes_written_slow": 0,
"max_bytes_wal": 0,
"max_bytes_db": 60438872064,
"max_bytes_slow": 0,
"alloc_unit_slow": 65536,
"alloc_unit_db": 1048576,
"alloc_unit_wal": 0,
"read_random_lat": {
"avgcount": 769669,
"sum": 47.340726985,
"avgtime": 6.1507e-05
},
"read_random_count": 769669,
"read_random_bytes": 26495066163,
"read_random_disk_count": 254435,
"read_random_disk_bytes": 24946337237,
"read_random_disk_bytes_wal": 0,
"read_random_disk_bytes_db": 24668980024,
"read_random_disk_bytes_slow": 277357213,
"read_random_buffer_count": 517401,
"read_random_buffer_bytes": 1548728926,
"read_lat": {
"avgcount": 169726,
"sum": 8.978258479,
"avgtime": 5.2898e-05
},
"read_count": 169726,
"read_bytes": 5083418859,
"read_disk_count": 113428,
"read_disk_bytes": 4156133376,
"read_disk_bytes_wal": 0,
"read_disk_bytes_db": 3756879872,
"read_disk_bytes_slow": 399257600,
"read_prefetch_count": 145957,
"read_prefetch_bytes": 4303778505,
"write_count": 14579621,
"write_disk_count": 14581706,
"write_bytes": 233965318144,
"compact_lat": {
"avgcount": 2029,
"sum": 20.610673612,
"avgtime": 0.010158045
},
"compact_lock_lat": {
"avgcount": 2029,
"sum": 1.662599252,
"avgtime": 0.000819418
},
"fsync_lat": {
"avgcount": 7268257,
"sum": 3465.742834128,
"avgtime": 0.000476832
},
"flush_lat": {
"avgcount": 7306924,
"sum": 2073.896410872,
"avgtime": 0.000283826
},
"unlink_lat": {
"avgcount": 10587,
"sum": 0.140086417,
"avgtime": 1.3231e-05
},
"truncate_lat": {
"avgcount": 1,
"sum": 1.152e-06,
"avgtime": 1.152e-06
},
"alloc_slow_fallback": 0,
"alloc_slow_size_fallback": 0,
"read_zeros_candidate": 0,
"read_zeros_errors": 0,
"wal_alloc_lat": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"db_alloc_lat": {
"avgcount": 24562,
"sum": 0.325040801,
"avgtime": 1.3233e-05
},
"slow_alloc_lat": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"alloc_wal_max_lat": 0,
"alloc_db_max_lat": 0.000259603,
"alloc_slow_max_lat": 0
},
"rocksdb": {
"get_latency": {
"avgcount": 1922412,
"sum": 88.397899767,
"avgtime": 4.5982e-05
},
"submit_latency": {
"avgcount": 9558177,
"sum": 800.936557968,
"avgtime": 8.3795e-05
},
"submit_sync_latency": {
"avgcount": 7246908,
"sum": 3558.578466937,
"avgtime": 0.000491047
},
"compact": 0,
"compact_running": 0,
"compact_completed": 0,
"compact_lasted": 0,
"compact_queue_merge": 0,
"compact_queue_len": 0,
"rocksdb_write_wal_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_memtable_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_delay_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
},
"rocksdb_write_pre_and_post_time": {
"avgcount": 0,
"sum": 0,
"avgtime": 0
}
}
}