Ceph: Increasing pg/pgp_num from 1024->2048

[John.N]

Hello,

I recently passed 41 OSDs on our cluster, and the PG calc suggests we should be at 2048 pgs now.
Also, I'm going to be adding 5 OSDs soon too.

I've read some people saying that it's best to increase pg_num in increments of 256.

Has anyone got experience in increasing from 1024 to 2048 or maybe more, and what do you guys think?


Thanks!

 

[Alwin]

Depending on your Ceph version, the autoscaler could be of use. The increase will happen automatically in the background.
https://docs.ceph.com/en/nautilus/rados/operations/placement-groups/

On the other hand when you set the new PG count Ceph will start to create the PGs and then rebalance/recover the data. Might be more resource intensive then the autoscaler, you may use osd_max_backfills or others to tune that process. Depending on your hardware of course.

 

[John.N]

Hello @Alwin , Luminous has no autoscaler AFAIK.

I have osd_max_backfills set to '1', so I'm thinking of just increasing pg_num , let it run slowly (for hours probably) and then pgp_num.

What do you think?

 

[RokaKen]

What do you think?

That will work; however, if this is a production cluster and you want the VMs/CTs to remain responsive, you may want to look at my post to a related thread which applies to Ceph Luminous (PVE 5.x)

 

[John.N]

Thank you for your link @RokaKen .
I'm running with 2x10G LAG and SSD disks (not NVMe), do you think that going up in 128 increments and with backfill set to '1' be OK for my users, as I want minimal impact on performance?

Of course it's going to run in the least busy time of day.

 

[John.N]

Just an update, went from 1024->2048 in 128 increments.
Everything went smoothly, thank you @RokaKen !

 

[eliyahuadam]

Hello John,

Can you share please the procedure (commands) for increasing the PG from 1024 to 2048?

Thanks.