Ceph PGs <75 or >200 per OSD ist just a warning?

[Knuuut]

Hi,

I'm going to migrate our cluster from HDDs to SSDs and from filestore with SSD-journal to bluestore. Not a big deal with plenty of time...

Unfortunately the pg_num was set to 1024 with 18 OSDs. Afaik this is not a good value, because if one node with 6 OSDs fails, the cluster will be unable to recover completely because the remaining 12 OSDs would be overloaded with 256 PGs/OSD.

My solution for this scenario would be adding an additional node with more OSDs, creating an additional Pool with less pg_num, copy images from VMs to the new Pool, destroy the old Pool. But I have to take care not running into less than 75 PGs/OSD after destroying the old Pool. I've done my math to calculate the pg_num for the new Pool witch fits inbetween 75 - 200 PGs per OSD.

But I wonder if it's safe to temporary step over or under these borders? Cephs gives me a warning, sure, but can I work a limitest amount of time under 75 or over 200 PGs? Maybe there will be a big performance impact in this scenario?

Thanks

Knuuut

 

[Alwin]

The target PG count is 100 for each OSD, 200 only if your cluster will expand to double its current size. Did you do the calculation with pgcalc? https://ceph.com/pgcalc/

As a temporary measure, you can set the limits/warnings for PGs per OSDs higher. But I advise to set it back to its defaults, after you finished your migratoin. http://docs.ceph.com/docs/luminous/rados/configuration/pool-pg-config-ref/

More information about the overdose protection.
https://ceph.com/community/new-luminous-pg-overdose-protection/

 

[Knuuut]

Did you do the calculation with pgcalc? https://ceph.com/pgcalc/

I'm aware of the calculation. Unfortunately the pg_num of 1024 has been set by someone else. So, this is the situation I'm dealing with.