CEPH Inactive PG - VM Little to no activity

[watnow101]

Hi

I had an issue today where one of my disks was busy failing with high latency of over 1000ms. I immediately stopped the disk and out it. However, after that a handful VM's was non responding. I saw under Ceph Health that there were 2 inactive pg's and only after a hour or so they disappeared and only then the VM's was responding again.






I never had this issue before and want to find out if something like this happens again, is there a way I can fix this quicker than waiting for it to clear?

 

[aaron]

The "low space hindering backfill" warning is ominous.

How full are the OSDs? Can you post the output of ceph osd df tree?

 

[watnow101]

ID  CLASS WEIGHT   TYPE NAME      STATUS REWEIGHT PRI-AFF
 -1       38.56575 root default                           
 -3        3.27475     host c3n1                         
  0   hdd  0.54579         osd.0      up  0.81169 1.00000
  1   hdd  0.54579         osd.1      up  1.00000 1.00000
  2   hdd  0.54579         osd.2      up  1.00000 1.00000
  3   hdd  0.54579         osd.3      up  1.00000 1.00000
  4   hdd  0.54579         osd.4      up  1.00000 1.00000
  5   hdd  0.54579         osd.5      up  1.00000 1.00000
 -5        3.27475     host c3n2                         
  6   hdd  0.54579         osd.6      up  0.95001 1.00000
  7   hdd  0.54579         osd.7      up  0.90002 1.00000
  8   hdd  0.54579         osd.8      up  1.00000 1.00000
  9   hdd  0.54579         osd.9      up  1.00000 1.00000
 10   hdd  0.54579         osd.10     up  0.90002 1.00000
 11   hdd  0.54579         osd.11     up  0.95001 1.00000
 -7        3.27472     host c3n3                         
 12   hdd  0.54578         osd.12     up  0.75006 1.00000
 13   hdd  0.54579         osd.13     up  0.90002 1.00000
 14   hdd  0.54579         osd.14     up  0.85002 1.00000
 15   hdd  0.54579         osd.15     up  0.90002 1.00000
 16   hdd  0.54579         osd.16     up  1.00000 1.00000
 17   hdd  0.54578         osd.17     up  0.75006 1.00000
 -9        2.57968     host c3n4                         
 21   hdd  0.90970         osd.21     up  0.95001 1.00000
 22   hdd  0.87000         osd.22     up  0.90639 1.00000
 23   hdd  0.79999         osd.23     up  1.00000 1.00000
-11        3.63879     host c3n5                         
 18   hdd  0.90970         osd.18     up  0.90002 1.00000
 20   hdd  0.90970         osd.20     up  1.00000 1.00000
 26   hdd  0.90970         osd.26     up  1.00000 1.00000
 27   hdd  0.90970         osd.27     up  1.00000 1.00000
-13        3.63879     host c3n6                         
 19   hdd  0.90970         osd.19     up  1.00000 1.00000
 28   hdd  0.90970         osd.28     up  0.95001 1.00000
 29   hdd  0.90970         osd.29     up  0.95001 1.00000
 34   hdd  0.90970         osd.34     up  1.00000 1.00000
-15        4.54849     host c3n7                         
 30   hdd  0.90970         osd.30     up  0.85004 1.00000
 31   hdd  0.90970         osd.31     up  1.00000 1.00000
 32   hdd  0.90970         osd.32     up  1.00000 1.00000
 33   hdd  0.90970         osd.33     up  0.95001 1.00000
 41   hdd  0.90970         osd.41     up  1.00000 1.00000
-17        7.27759     host c3n8                         
 24   hdd  1.81940         osd.24   down        0 1.00000
 25   hdd  1.81940         osd.25     up  1.00000 1.00000
 35   hdd  1.81940         osd.35     up  1.00000 1.00000
 39   hdd  1.81940         osd.39     up  1.00000 1.00000
-19        7.05818     host c3n9                         
 36   hdd  1.81940         osd.36   down        0 1.00000
 37   hdd  1.81940         osd.37     up  1.00000 1.00000
 38   hdd  1.59999         osd.38     up  1.00000 1.00000
 40   hdd  1.81940         osd.40     up  1.00000 1.00000

 

[watnow101]

Average OSD size is between 60% and 70% - with the odd one or two that grows above 80%.