High Latency on a new HDD-disks (ceph)

[DEZERTIR]

Hello

We installed nine new HDD drives in a CEPH cluster, a Proxmox VE cluster.

For a week now, we've been observing that only these drives have high latency (20-80).

There's no PG imbalance; everything is fine with the old HDD drives.

View attachment 99016

 

[DEZERTIR]

CRUSH map:

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class ssd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class ssd
device 7 osd.7 class ssd
device 8 osd.8 class ssd
device 9 osd.9 class ssd
device 10 osd.10 class ssd
device 11 osd.11 class hdd
device 12 osd.12 class hdd
device 13 osd.13 class hdd
device 14 osd.14 class hdd
device 15 osd.15 class hdd
device 16 osd.16 class hdd
device 17 osd.17 class hdd
device 18 osd.18 class hdd
device 19 osd.19 class hdd
device 20 osd.20 class hdd
device 21 osd.21 class hdd
device 22 osd.22 class hdd
device 23 osd.23 class hdd
device 24 osd.24 class hdd
device 25 osd.25 class hdd
device 26 osd.26 class hdd
device 27 osd.27 class ssd
device 28 osd.28 class nvme
device 29 osd.29 class nvme
device 30 osd.30 class nvme
device 31 osd.31 class nvme
device 32 osd.32 class nvme
device 33 osd.33 class nvme
device 34 osd.34 class nvme
device 35 osd.35 class nvme
device 36 osd.36 class nvme
device 37 osd.37 class nvme
device 38 osd.38 class nvme
device 39 osd.39 class nvme
device 40 osd.40 class hdd
device 41 osd.41 class hdd
device 42 osd.42 class hdd
device 43 osd.43 class hdd
device 44 osd.44 class nvme
device 45 osd.45 class nvme
device 46 osd.46 class nvme
device 47 osd.47 class nvme
device 48 osd.48 class nvme
device 49 osd.49 class nvme
device 50 osd.50 class ssd
device 51 osd.51 class ssd
device 52 osd.52 class ssd
device 53 osd.53 class ssd
device 54 osd.54 class ssd
device 55 osd.55 class ssd
device 56 osd.56 class ssd
device 57 osd.57 class ssd
device 58 osd.58 class ssd
device 59 osd.59 class nvme
device 60 osd.60 class nvme
device 61 osd.61 class nvme
device 62 osd.62 class ssd
device 63 osd.63 class ssd
device 64 osd.64 class ssd
device 65 osd.65 class hdd
device 66 osd.66 class hdd
device 67 osd.67 class hdd
device 68 osd.68 class hdd
device 69 osd.69 class hdd
device 70 osd.70 class nvme
device 71 osd.71 class nvme
device 72 osd.72 class nvme
device 73 osd.73 class ssd
device 74 osd.74 class ssd
device 75 osd.75 class ssd
device 76 osd.76 class hdd
device 77 osd.77 class hdd
device 78 osd.78 class hdd
device 79 osd.79 class hdd
device 80 osd.80 class hdd
device 81 osd.81 class hdd
device 82 osd.82 class hdd
device 83 osd.83 class hdd
device 84 osd.84 class hdd
device 85 osd.85 class hdd
device 86 osd.86 class hdd
device 87 osd.87 class ssd
device 88 osd.88 class ssd
device 89 osd.89 class ssd
device 90 osd.90 class ssd
device 91 osd.91 class ssd
device 92 osd.92 class ssd
device 93 osd.93 class ssd
device 94 osd.94 class ssd
device 95 osd.95 class ssd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root

# buckets
host tvr-pve-04 {
id -3 # do not change unnecessarily
id -2 class ssd # do not change unnecessarily
id -11 class hdd # do not change unnecessarily
id -16 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.3 weight 6.98630
item osd.4 weight 6.98630
item osd.5 weight 6.98630
item osd.15 weight 9.09569
item osd.16 weight 9.09569
item osd.17 weight 9.09569
item osd.18 weight 9.09569
item osd.30 weight 2.91100
item osd.31 weight 2.91100
item osd.38 weight 2.91100
item osd.42 weight 9.09569
item osd.83 weight 9.09569
item osd.90 weight 6.98630
}
host tvr-pve-03 {
id -5 # do not change unnecessarily
id -6 class ssd # do not change unnecessarily
id -12 class hdd # do not change unnecessarily
id -17 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.0 weight 6.98630
item osd.1 weight 6.98630
item osd.2 weight 6.98630
item osd.11 weight 9.09569
item osd.13 weight 9.09569
item osd.32 weight 2.91100
item osd.33 weight 2.91100
item osd.39 weight 2.91100
item osd.43 weight 9.09569
item osd.14 weight 9.09569
item osd.12 weight 9.09569
item osd.84 weight 9.09569
item osd.89 weight 6.98630
}
host tvr-pve-05 {
id -7 # do not change unnecessarily
id -8 class ssd # do not change unnecessarily
id -13 class hdd # do not change unnecessarily
id -18 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.9 weight 6.98630
item osd.10 weight 6.98630
item osd.27 weight 6.98630
item osd.19 weight 9.09569
item osd.20 weight 9.09569
item osd.21 weight 9.09569
item osd.22 weight 9.09569
item osd.28 weight 2.91100
item osd.29 weight 2.91100
item osd.37 weight 2.91100
item osd.41 weight 9.09569
item osd.82 weight 9.09569
item osd.91 weight 6.98630
}
host tvr-pve-06 {
id -9 # do not change unnecessarily
id -10 class ssd # do not change unnecessarily
id -14 class hdd # do not change unnecessarily
id -19 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.6 weight 6.98630
item osd.7 weight 6.98630
item osd.8 weight 6.98630
item osd.23 weight 9.09569
item osd.24 weight 9.09569
item osd.25 weight 9.09569
item osd.26 weight 9.09569
item osd.34 weight 2.91100
item osd.35 weight 2.91100
item osd.36 weight 2.91100
item osd.40 weight 9.09569
item osd.81 weight 9.09569
item osd.92 weight 6.98630
}
host tvr-pve-07 {
id -21 # do not change unnecessarily
id -22 class ssd # do not change unnecessarily
id -23 class hdd # do not change unnecessarily
id -24 class nvme # do not change unnecessarily
# weight 36.67818
alg straw2
hash 0 # rjenkins1
item osd.44 weight 2.91100
item osd.45 weight 2.91100
item osd.46 weight 2.91100
item osd.50 weight 6.98630
item osd.51 weight 6.98630
item osd.54 weight 6.98630
item osd.93 weight 6.98630
}
host tvr-pve-08 {
id -25 # do not change unnecessarily
id -26 class ssd # do not change unnecessarily
id -27 class hdd # do not change unnecessarily
id -28 class nvme # do not change unnecessarily
# weight 36.67818
alg straw2
hash 0 # rjenkins1
item osd.47 weight 2.91100
item osd.48 weight 2.91100
item osd.49 weight 2.91100
item osd.52 weight 6.98630
item osd.53 weight 6.98630
item osd.55 weight 6.98630
item osd.94 weight 6.98630
}
host tvr-pve-09 {
id -29 # do not change unnecessarily
id -30 class ssd # do not change unnecessarily
id -31 class hdd # do not change unnecessarily
id -32 class nvme # do not change unnecessarily
# weight 27.94519
alg straw2
hash 0 # rjenkins1
item osd.56 weight 6.98630
item osd.57 weight 6.98630
item osd.58 weight 6.98630
item osd.95 weight 6.98630
}
host tvr-pve-02 {
id -33 # do not change unnecessarily
id -34 class ssd # do not change unnecessarily
id -35 class hdd # do not change unnecessarily
id -36 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.59 weight 2.91100
item osd.60 weight 2.91100
item osd.61 weight 2.91100
item osd.62 weight 6.98630
item osd.63 weight 6.98630
item osd.64 weight 6.98630
item osd.65 weight 9.09569
item osd.66 weight 9.09569
item osd.67 weight 9.09569
item osd.68 weight 9.09569
item osd.69 weight 9.09569
item osd.85 weight 9.09569
item osd.88 weight 6.98630
}
host tvr-pve-01 {
id -37 # do not change unnecessarily
id -38 class ssd # do not change unnecessarily
id -39 class hdd # do not change unnecessarily
id -40 class nvme # do not change unnecessarily
# weight 91.25230
alg straw2
hash 0 # rjenkins1
item osd.70 weight 2.91100
item osd.71 weight 2.91100
item osd.72 weight 2.91100
item osd.73 weight 6.98630
item osd.74 weight 6.98630
item osd.75 weight 6.98630
item osd.76 weight 9.09569
item osd.77 weight 9.09569
item osd.78 weight 9.09569
item osd.79 weight 9.09569
item osd.80 weight 9.09569
item osd.86 weight 9.09569
item osd.87 weight 6.98630
}
root default {
id -1 # do not change unnecessarily
id -4 class ssd # do not change unnecessarily
id -15 class hdd # do not change unnecessarily
id -20 class nvme # do not change unnecessarily
# weight 648.81537
alg straw2
hash 0 # rjenkins1
item tvr-pve-04 weight 91.25230
item tvr-pve-03 weight 91.25230
item tvr-pve-05 weight 91.25230
item tvr-pve-06 weight 91.25230
item tvr-pve-07 weight 36.67818
item tvr-pve-08 weight 36.67818
item tvr-pve-09 weight 27.94519
item tvr-pve-02 weight 91.25230
item tvr-pve-01 weight 91.25230
}

# rules
rule replicated_rule {
id 0
type replicated
step take default
step chooseleaf firstn 0 type host
step emit
}
rule replicated_hdd {
id 1
type replicated
step take default class hdd
step chooseleaf firstn 0 type host
step emit
}
rule replicated_ssd {
id 2
type replicated
step take default class ssd
step chooseleaf firstn 0 type host
step emit
}
rule replicated_nvme {
id 3
type replicated
step take default class nvme
step chooseleaf firstn 0 type host
step emit
}

# end crush map
Server View
Logs

 

[DEZERTIR]

HDD-disks 10TB SAS 12Gb/s Seagate
ST10000NM013G Exos X18 3.5" 7200rpm 256MB

 

[guruevi]

Are they seated well in an appropriate bay with shock mounts? High latency specific to a spinning drive could be due to vibrations. Are they formatted to their 4Kn, because they can also be formatted to 512b sector size which would affect performance.

 

[alexskysilk]

Latency in the 80-100 range is normal. Just means the drives are busy. are they currently rebalancing?

 

[Nemesiz]

Try to disable write-cache for HDD

 

[JoaGamo]

As already said, 80-100ms latency is very normal for HDDs

If you want to improve Ceph performance on the HDD pool, consider offloading WAL + RocksDB to a SSD for HDD OSDs.
The safest method implies you have to evacuate each existing HDD OSD (draining all PGs off it) and re-create the HDD OSD with the WAL+DB volume offloaded to a SSD.

If the drive hosting the WAL + DB dies, all OSDs linked to it will die, so if you plan on sharing the same SSD across multiple HDD OSDs, I recommend configuring the SSD as a mirrored pair.