After last update today i get "too many PGs per OSD (256 > max 200)"

[hape]

Hello all,

after the latest update, done today in the morning, i get a HEALTH_WARN in Ceph

-> "too many PGs per OSD (256 > max 200)"

Before the update the HEALTH of Ceph was well.

Are there any changes in CEPH-stuff, or have i done anything wrong in my setup?

 

[fabian]

see the Ceph release notes for 12.2.1 (the change was actually only contained in 12.2.2, the release note change was merged too early).

TL;DR: the limit was decreased (from 300 to 200) and changed from "warn when over this limit" to "warn when over this limit and prevent changing the PG count over the limit".

 

[hape]

Should i create a new pool with the correct limit?

Do i have to switch the VMs to a new created pool or can i correct it in the running pool?

 

[Alwin]

You can't decrease the number of PGs only increase. Check out the blog post, it pretty much covers every step for such a procedure.
https://swamireddy.wordpress.com/2017/04/18/ceph-reduce-the-pg-number-on-a-pool/

 

[hape]

Hello,

by the way i'm not able to create a new pool with correct values. The message i get is:

mon_command failed - pg_num 64 size 3 would mean 960 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)@çøU

Am i wrong to do this?

 

[Alwin]

ceph osd df tree

With the above command you can see how many PGs are on the OSDs. As a traget, your OSDs should be close to 100 PGs, 200 is if your cluster will expand at least double in size. To protect against too many PGs per OSD this limit is enforced.

You can now add more OSDs to the cluster to reduce the number of PGs/OSD. Move your data onto a different storage, delete the pool and recreate it. Or override the hard limit to go on.

 

[Rob Loan]

Thanks for the `ceph osd df tree` command. Went from a pg_num=512 assumed from the ceph calculator

ID CLASS WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE VAR  PGS TYPE NAME       
-1       5.45755        - 5588G 86552M 5503G 1.51 1.00   - root default   
-5       1.81918        - 1862G 28875M 1834G 1.51 1.00   -     host MacPro
 2   hdd 0.90959  1.00000  931G 14105M  917G 1.48 0.98 255         osd.2   
 3   hdd 0.90959  1.00000  931G 14770M  916G 1.55 1.02 257         osd.3   
-3       1.81918        - 1862G 28841M 1834G 1.51 1.00   -     host Super 
 0   hdd 0.90959  1.00000  931G 14767M  916G 1.55 1.02 259         osd.0   
 1   hdd 0.90959  1.00000  931G 14074M  917G 1.48 0.98 253         osd.1   
-7       1.81918        - 1862G 28835M 1834G 1.51 1.00   -     host nas   
 4   hdd 0.90959  1.00000  931G 14196M  917G 1.49 0.98 253         osd.4   
 5   hdd 0.90959  1.00000  931G 14639M  917G 1.53 1.01 259         osd.5   
                    TOTAL 5588G 86552M 5503G 1.51                         
MIN/MAX VAR: 0.98/1.02  STDDEV: 0.03

to pg_num=128 so I have room for a second pool to stay under 100 PGs per 6 OSDs

ID CLASS WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE VAR  PGS TYPE NAME       
-1       5.45755        - 5588G 72462M 5517G 1.27 1.00   - root default   
-5       1.81918        - 1862G 24231M 1839G 1.27 1.00   -     host MacPro
 2   hdd 0.90959  1.00000  931G 14029M  917G 1.47 1.16  75         osd.2   
 3   hdd 0.90959  1.00000  931G 10202M  921G 1.07 0.84  53         osd.3   
-3       1.81918        - 1862G 24104M 1839G 1.26 1.00   -     host Super 
 0   hdd 0.90959  1.00000  931G 13735M  918G 1.44 1.14  75         osd.0   
 1   hdd 0.90959  1.00000  931G 10369M  921G 1.09 0.86  53         osd.1   
-7       1.81918        - 1862G 24126M 1839G 1.26 1.00   -     host nas   
 4   hdd 0.90959  1.00000  931G 11485M  920G 1.20 0.95  60         osd.4   
 5   hdd 0.90959  1.00000  931G 12641M  919G 1.33 1.05  68         osd.5   
                    TOTAL 5588G 72462M 5517G 1.27                         
MIN/MAX VAR: 0.84/1.16  STDDEV: 0.16

 

[Alwin]

I have room for a second pool to stay under 100 PGs per 6 OSDs

Well, you should target around 100 PGs per OSD, as this is not a power of 2, you might be ending up with 128. The PGcalc should show how many PGs you may need.
http://ceph.com/pgcalc/

 

[hape]

After moving VMs to a different storage an delete the pool and creating a new one with pg_num=64 all is running fine and the HEALTH is OK ;-)

 

[ddkargas]

12.2.2

In Global Session

mon_max_pg_per_osd = 300
osd_max_pg_per_osd_hard_ratio = 1.2

After Restart Target now is Health ok
I don't now if this correct but is ok now

 

[Alwin]

@ddkargas, you overrule the settings to the old values. These have been changed to lower the overload of systems, when too many PGs need to be calculated per OSD.

 

[ddkargas]

@Alwin
I can not say that I understand exactly what you mean, I'm very young at the ceph,
I've planned to add some osd demons even think that will solve the problem.
Thanks

 

[Alwin]

Please do so, as the target count for each OSD should be around 100 PGs (a power of two). Raising the max limit is only for ceph cluster that are running with the old settings (have changed with luminous).

 

[ddkargas]

h..p://technik.blogs.nde.ag/2017/12/26/ceph-12-2-2-minor-update-major-trouble/
I read this article and tried to figure it out.

 

[tuxis]

Please note that setting this value in the global section is sufficient:

mon_max_pg_per_osd = 300

After setting this, you need to restart the mgr-daemons to get rid of the error.

Obviously, adding more OSD's is the better solution, but that's not always possible on a production cluster.

 

[ddkargas]

Thanks @tuxis it is ok now, you are right this parameter is ok.