Resolving "full osd(s)", "backfillfull osd(s)" and "pool(s) full"

[bluewolf42]

Yesterday I was greeted with numerous unreachable services stemming from a Ceph health error on our VM cluster due to "1 full osd(s)" and "1 backfillfull osd(s)" resulting in "4 pool(s) full" and solved it. This was the ceph status panel:



As a result, our Ceph ended up in read-only state. The full OSDs were shown directly in the details of the status summary. After some reading and looking at our configuration I got it fixed by doing the following:

1. Increase the full and backfillfull ratios a little such that Ceph has some space to work with and switches from a health error to a health warning:
   

   sudo ceph osd set-backfillfull-ratio 0.93
   sudo ceph osd set-full-ratio 0.97

2. The Balancer was running in mode upmap, but sudo ceph osd df showed a great imbalance in utilization of our ~50 OSDs. The solution was to set require-min-compat-client to luminous such that the balancer is able to perform the upmap mode properly as noted in the corresponding ceph docs:
   

   sudo ceph osd set-require-min-compat-client luminous

   Here's more info about the balancer itself.
3. After an hour, everything was rebalanced and ceph in a healthy state again \o/. I could revert the backfillfull- and full-ratios to their previous values:
   

   sudo ceph osd set-backfillfull-ratio 0.90
   sudo ceph osd set-full-ratio 0.95

4. Some VMs ended up in a strange state but luckily worked normally after a reboot.

I still have a few questions though, maybe you can help:

1. The balancer seemed to work at least partially, otherwise this problem would have happened sooner I guess. Is it intended that it is able to run in upmap mode together with require-min-compat-client set to something pre-luminous (was "jewel" in our case) although the docs say: "supporting only Luminous (and newer) clients"?
2. In the PVE upgrade instructions from Ceph Jewel to Luminous this setting is explicitly mentioned. However, there it is adviced to set this value to jewel instead of luminous - is that indended or a typo? I didn't find set-require-min-compat-client anywhere else in the PVE wiki.

If my solution wasn't quite textbook, feel free to jump in and suggest better ones =)

Thanks in advance