Slow ops in ceph.

bestof

New Member
Aug 1, 2023
4
0
1
Hi, i have 2 erros regarding ceph.
I have ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable) on all nodes.
1- Reduced data availability: 128 pgs inactive, 5 pgs stale:
pg 1.8 is stuck stale for 2d, current state stale+active+clean, last acting [1,2,3]
pg 1.a is stuck stale for 2d, current state stale+active+clean, last acting [1,2,0]
pg 5.3 is stuck inactive for 3d, current state unknown, last acting []
pg 5.c is stuck inactive for 3d, current state unknown, last acting []
pg 5.d is stuck inactive for 3d, current state unknown, last acting [] ...

2- 10 slow ops, oldest one blocked for 4228 sec, daemons [mon.proxmox-a,mon.proxmox-f,mon.proxmox-g,mon.proxmox-h] have slow ops.

If i try to query the pg: - for the first 2 is ok i get the output but for all the rest i get
Code:
ceph pg 5.3 query
Couldn't parse JSON : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/bin/ceph", line 1326, in <module>
    retval = main()
  File "/usr/bin/ceph", line 1246, in main
    sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 993, in parse_json_funcsigs
    raise e
  File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 990, in parse_json_funcsigs
    overall = json.loads(s)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
this are the detalis
Code:
ceph osd tree
ID   CLASS  WEIGHT    TYPE NAME           STATUS  REWEIGHT  PRI-AFF
 -1         31.87491  root default                                
-19          3.63879      host proxmox-a                          
 10   nvme   1.81940          osd.10          up   1.00000  1.00000
 11   nvme   1.81940          osd.11          up   1.00000  1.00000
-16          3.63879      host proxmox-b                          
  8   nvme   1.81940          osd.8           up   1.00000  1.00000
  9   nvme   1.81940          osd.9           up   1.00000  1.00000
-13          3.63879      host proxmox-e                          
  6   nvme   1.81940          osd.6           up   1.00000  1.00000
  7   nvme   1.81940          osd.7           up   1.00000  1.00000
 -3          6.98618      host proxmox-f                          
  0    ssd   3.49309          osd.0           up   1.00000  1.00000
  3    ssd   3.49309          osd.3           up   1.00000  1.00000
 -5          6.98618      host proxmox-g                          
  1    ssd   3.49309          osd.1           up   1.00000  1.00000
  4    ssd   3.49309          osd.4           up   1.00000  1.00000
 -7          6.98618      host proxmox-h                          
  2    ssd   3.49309          osd.2           up   1.00000  1.00000
  5    ssd   3.49309          osd.5           up   1.00000  1.00000

Code:
ceph osd lspools
1 ssd_pool
2 .mgr
5 nvme_pool

Code:
ceph -s
  cluster:
    id:     xxxxxx
    health: HEALTH_WARN
            Reduced data availability: 128 pgs inactive, 5 pgs stale
            10 slow ops, oldest one blocked for 4228 sec, daemons [mon.proxmox-a,mon.proxmox-f,mon.proxmox-g,mon.proxmox-h] have slow ops.
 
  services:
    mon: 6 daemons, quorum proxmox-g,proxmox-h,proxmox-f,proxmox-a,proxmox-b,proxmox-e (age 14m)
    mgr: proxmox-h(active, since 3d), standbys: proxmox-f, proxmox-e, proxmox-g, proxmox-a, proxmox-b
    osd: 12 osds: 12 up (since 3h), 12 in (since 2w)
 
  data:
    pools:   3 pools, 145 pgs
    objects: 1.12M objects, 3.6 TiB
    usage:   11 TiB used, 21 TiB / 32 TiB avail
    pgs:     88.276% pgs unknown
             128 unknown
             12  active+clean
             5   stale+active+clean

Cpeh is very slow and some commands i dont get any output anymore, is frozen. What can i do ?
 
Last edited:
I did it but nothing changed
Code:
systemctl restart ceph-mon@proxmox-a.service

systemctl restart ceph-mon@proxmox-b.service

systemctl restart ceph-mon@proxmox-e.service

systemctl restart ceph-mon@proxmox-f.service

systemctl restart ceph-mon@proxmox-g.service

systemctl restart ceph-mon@proxmox-h.service

Code:
ceph health detail
HEALTH_WARN Reduced data availability: 128 pgs inactive, 5 pgs stale; 10 slow ops, oldest one blocked for 4228 sec, daemons [mon.proxmox-a,mon.proxmox-f,mon.proxmox-g,mon.proxmox-h] have slow ops.
[WRN] PG_AVAILABILITY: Reduced data availability: 128 pgs inactive, 5 pgs stale
    pg 1.8 is stuck stale for 3d, current state stale+active+clean, last acting [1,2,3]
    pg 1.a is stuck stale for 3d, current state stale+active+clean, last acting [1,2,0]
    pg 5.3 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.c is stuck inactive for 3d, current state unknown, last acting []
    pg 5.d is stuck inactive for 3d, current state unknown, last acting []
    pg 5.e is stuck inactive for 3d, current state unknown, last acting []
    pg 5.f is stuck inactive for 3d, current state unknown, last acting []
    pg 5.10 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.11 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.12 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.13 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.14 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.15 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.16 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.17 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.18 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.19 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1a is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1b is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1c is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1d is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1e is stuck inactive for 3d, current state unknown, last acting []
    pg 5.1f is stuck inactive for 3d, current state unknown, last acting []
    pg 5.20 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.21 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.22 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.23 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.24 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.25 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.26 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.27 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.28 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.29 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.6e is stuck inactive for 3d, current state unknown, last acting []
    pg 5.6f is stuck inactive for 3d, current state unknown, last acting []
    pg 5.70 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.71 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.72 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.73 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.74 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.75 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.76 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.77 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.78 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.79 is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7a is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7b is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7c is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7d is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7e is stuck inactive for 3d, current state unknown, last acting []
    pg 5.7f is stuck inactive for 3d, current state unknown, last acting []
[WRN] SLOW_OPS: 10 slow ops, oldest one blocked for 4228 sec, daemons [mon.proxmox-a,mon.proxmox-f,mon.proxmox-g,mon.proxmox-h] have slow ops.
 
ok,but i don't think the problem is from there, it was ok for the last 2 years. Also Ceph is very slow.
 
Hi, sorry for the late answer and thank you for your time. Is seems to be ok form ths disk size, th store.db is just 12% occupying.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!