Increase pg_num on running ceph pool

[Shane Poppleton]

I am running a proxmox ve cluster, with currently 8 nodes, 2 x 1TB OSD's per node. On my ceph performance tab it says that my usage is 3.2Tb of 8.95Tb. But if i check the shared storage it says 1.1Tb used of 2.68Tb.

I can't see anywhere where I could change the pool size. pg_num on my storage pool is currently set to 128, and I believe I may need to increase the size of that, are the size of the pool related to the pg_num or is that completely different.

I also read that a cluster of this size (8 nodes) should have a larger pg_num, but have not been able to find how to change this.

 

[badji]

#ceph osd lspools
#ceph osd pool get <your-ceph-storage> pg_num

and to change it :
#ceph osd pool set <your-ceph-storage> pg_num <new-number>

 

[Shane Poppleton]

will increasing the pg_num, increase the capacity of the pool, if not, how would i do that?

 

[alexskysilk]

will increasing the pg_num, increase the capacity of the pool

No

if not, how would i do that?

Add more OSDS, replace OSDs with larger disks, and/or change your pool allocation rules.

 

[RokaKen]

@Shane Poppleton Keep in mind when increasing pg_num, you must also increase pgp_num to actually use the increased PGs. If doing this during production with clients up, you should increase both in small increments (waiting for the rebalance to complete between each) until the target PG count is reached.

 

[Shane Poppleton]

OK, I tried increasing pg_num with the command specified and got the following message:

Error E2BIG: The specified pg_num 512 is too large (creating 384 new PGs on 10 OSDs exceeds per OSD max of 32)

 

[RokaKen]

So, you are seeing CEPH Luminous trying to prevent you from making drastic changes (Issue #39353). This is actually saying to you that going from 128 PGs directly to 512 PGs (384) exceeds the number of active OSDs (10) * mon_osd_max_split_count (32) = 320.

Just increase pg_num and pgp_num in smaller increments, e.g. 256, let it backfill and get all PGs to active+clean, then next increment up to 512 (or appropriate PG count).