Increase pg_num on running ceph pool

May 31, 2018
I am running a proxmox ve cluster, with currently 8 nodes, 2 x 1TB OSD's per node. On my ceph performance tab it says that my usage is 3.2Tb of 8.95Tb. But if i check the shared storage it says 1.1Tb used of 2.68Tb.

I can't see anywhere where I could change the pool size. pg_num on my storage pool is currently set to 128, and I believe I may need to increase the size of that, are the size of the pool related to the pg_num or is that completely different.

I also read that a cluster of this size (8 nodes) should have a larger pg_num, but have not been able to find how to change this.
#ceph osd lspools
#ceph osd pool get <your-ceph-storage> pg_num

and to change it :
#ceph osd pool set <your-ceph-storage> pg_num <new-number>
@Shane Poppleton Keep in mind when increasing pg_num, you must also increase pgp_num to actually use the increased PGs. If doing this during production with clients up, you should increase both in small increments (waiting for the rebalance to complete between each) until the target PG count is reached.
OK, I tried increasing pg_num with the command specified and got the following message:

Error E2BIG: The specified pg_num 512 is too large (creating 384 new PGs on 10 OSDs exceeds per OSD max of 32)
So, you are seeing CEPH Luminous trying to prevent you from making drastic changes (Issue #39353). This is actually saying to you that going from 128 PGs directly to 512 PGs (384) exceeds the number of active OSDs (10) * mon_osd_max_split_count (32) = 320.

Just increase pg_num and pgp_num in smaller increments, e.g. 256, let it backfill and get all PGs to active+clean, then next increment up to 512 (or appropriate PG count).


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!