BlueFS spillover very dispair on disks

[samumsq]

Hello everyone,
We have a proxmox cluster and we are using Ceph 19.2.2. Our cluster has 3 nodes, in each node we have an SSD that we use for all OSDs db, and 5 HDD.
Yesterday, we started to experience BlueFS spillover on only 2 OSDs. After checking we see that this 2 OSD (OSD 5 and OSD 8) use almost all db space, but all of the others barely use 5GB

ceph df detail ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 33.71567 - 34 TiB 20 TiB 19 TiB 17 MiB 261 GiB 14 TiB 58.01 1.00 - root default -3 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 63 GiB 4.7 TiB 57.97 1.00 - host pve01-poz 1 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 5.6 MiB 3.7 GiB 655 GiB 65.86 1.14 56 up osd.1 2 hdd 1.87309 1.00000 1.9 TiB 994 GiB 939 GiB 24 KiB 3.3 GiB 924 GiB 51.85 0.89 45 up osd.2 3 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 30 KiB 3.3 GiB 747 GiB 61.08 1.05 53 up osd.3 4 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 23 KiB 2.9 GiB 648 GiB 66.20 1.14 58 up osd.4 10 hdd 1.87309 1.00000 1.9 TiB 942 GiB 887 GiB 31 KiB 3.2 GiB 976 GiB 49.14 0.85 44 up osd.10 15 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 975 GiB 36 KiB 46 GiB 888 GiB 53.72 0.93 49 up osd.15 -5 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 175 GiB 4.7 TiB 58.09 1.00 - host pve02-poz 5 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 27 KiB 68 GiB 701 GiB 63.62 1.10 55 up osd.5 6 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 983 GiB 5.6 MiB 31 GiB 881 GiB 54.09 0.93 47 up osd.6 7 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 37 KiB 2.7 GiB 716 GiB 62.87 1.08 56 up osd.7 8 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 1013 GiB 23 KiB 57 GiB 848 GiB 55.79 0.96 48 up osd.8 9 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.1 TiB 46 KiB 2.9 GiB 776 GiB 59.54 1.03 52 up osd.9 16 hdd 1.87309 1.00000 1.9 TiB 1009 GiB 954 GiB 35 KiB 13 GiB 909 GiB 52.59 0.91 47 up osd.16 -7 11.23856 - 11 TiB 6.5 TiB 6.2 TiB 5.7 MiB 23 GiB 4.7 TiB 57.97 1.00 - host pve03-poz 0 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 25 KiB 3.1 GiB 794 GiB 58.62 1.01 48 up osd.0 11 hdd 1.87309 1.00000 1.9 TiB 1.0 TiB 971 GiB 18 KiB 3.5 GiB 892 GiB 53.48 0.92 48 up osd.11 12 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.1 TiB 5.6 MiB 3.0 GiB 735 GiB 61.67 1.06 55 up osd.12 13 hdd 1.87309 1.00000 1.9 TiB 1.1 TiB 1.0 TiB 39 KiB 2.9 GiB 824 GiB 57.06 0.98 49 up osd.13 14 hdd 1.87309 1.00000 1.9 TiB 1.2 TiB 1.2 TiB 25 KiB 3.7 GiB 653 GiB 65.96 1.14 60 up osd.14 17 hdd 1.87309 1.00000 1.9 TiB 979 GiB 924 GiB 18 KiB 6.9 GiB 939 GiB 51.05 0.88 45 up osd.17

Is this normal behaviour for Ceph? Should we check anything before buying a bigger SSD for OSDs DB?
Thanks in advance for your help

 

[Tmanok]

Hi samumsq!

There's quite a high meta data consumption in general from the output. That CEPH cluster is running on very few disks, it's intended for hundreds of disks so strange behaviour can occur in smaller populations. Taking a look at the meta data distribution, though:

OSD	META
5	68 GiB
8	57 GiB
15	46 GiB
6	31 GiB
16	13 GiB
most others	~3 GiB

You can clearly see that some OSDs are producing more, which points to an FS oddity. Looking at it from a per host perspective, you have:

• pve01: about 63 GiB META
• pve02: about 175 GiB META
• pve03: about 23 GiB META

Which is telling us that your second node has something going on with it. But don't take my word for it, the Squid Documentation outlined what normal DB usage looks like for your HDD OSDs (1%-3% size of disk), which at about 1.87 TiB each, the rough guidelines would say:

• 1% ≈ 19 GiB
• 2% ≈ 38 GiB
• 3% ≈ 56 GiB
• 4% ≈ 77 GiB

It's too opaque / lacking in detail for me to say what is wrong but I can see that something looks wrong between that list and the list above for pve02. Can you please run the following commands?

ceph health detail
ceph daemon osd.5 bluestore bluefs device info
ceph daemon osd.7 bluestore bluefs device info
ceph daemon osd.8 bluestore bluefs device info
ceph daemon osd.15 bluestore bluefs device info
ceph-volume lvm list
lvs -a -o vg_name,lv_name,lv_size,devices

bluefs device info output is useful here as it tells you how much space BlueFS actually has on BDEV_DB and the slow device rather than trying to infer DB occupancy from ceph df detail.

That's as much as I can help for now, gotta get back to my day job and finish another draft question!
Thanks,


Tmanok