Adding Different size OSD, Running out of Disk-space, what to look out for.

[JeroenLWD]

Hi All,

I have a 6 node cluster that is faster running out of space than expected.
Got core`s and memory enough,

My Ceph pool almost at 70% of used space
I got 7 SSD in all six nodes of 480 GB 2/3 Replication.
1 have one slot left in all Six nodes.

The wearout of the 480 GB disk is only 6% so replacing them all for biger one`s seems like trowing a way a lot of money.

That is why i was thinking of adding a bigger disk in the last slot a 960 GB SSD.
And when i am at the point again of needing more space replacing 1 480GB ssd per node for a biger one of 960 GB

I have read that the 960Gb ssd as having more space will get more read and write`s that a smaller disk.
And in my slow down a bit.
Having a newer type of ssd with a bit more iops than the 480Gb is having i hope in will not even notice a dip in performance


What should i take into Account more ? or change in my configuration.
To bring this slow upgrade to a su6 ;-)

 

[VictorSTS]

As the new drive will be twice as big as the old ones, it will get twice as much weight, thus making them potentially storing twice as much info than an smaller disk and getting potentially as much as twice I/O requests as one small drive.

That might not be an issue at all if your current average I/O load on the small disks times 2 is still lower than the performace of your new 960GB drive: the new drive won't bottleneck the performance of your ceph cluster.

Keep in mind that having 8 disks per ceph node, only those landing on PG's stored in the 960GB disk may get slower if that disk is overloaded (2/9 of the requests? I'm not that good with arithmetics ).

Also, you could reweight the new disks anytime to make the store less PGs, thus receiving less I/Os at the cost of not filling them up as much as they will with their default weight.

As an alternative, create a new device class, create a new crush rule that uses only disks with this new device class and create a new pool with this crush rule. Then you can migrate some VM's disks to the new pool which will only use 960GB disks. Oh, this will only work as shown if you are *not* using the default crush "replicated_rule", as this default rule does not split device classes.

You have a nice example here:

https://forum.proxmox.com/threads/ceph-classes-rules-and-pools.60013/

 

[JeroenLWD]

As the new drive will be twice as big as the old ones, it will get twice as much weight, thus making them potentially storing twice as much info than an smaller disk and getting potentially as much as twice I/O requests as one small drive.

That might not be an issue at all if your current average I/O load on the small disks times 2 is still lower than the performace of your new 960GB drive: the new drive won't bottleneck the performance of your ceph cluster.

Keep in mind that having 8 disks per ceph node, only those landing on PG's stored in the 960GB disk may get slower if that disk is overloaded (2/9 of the requests? I'm not that good with arithmetics ).

Also, you could reweight the new disks anytime to make the store less PGs, thus receiving less I/Os at the cost of not filling them up as much as they will with their default weight.

As an alternative, create a new device class, create a new crush rule that uses only disks with this new device class and create a new pool with this crush rule. Then you can migrate some VM's disks to the new pool which will only use 960GB disks. Oh, this will only work as shown if you are *not* using the default crush "replicated_rule", as this default rule does not split device classes.

You have a nice example here:

https://forum.proxmox.com/threads/ceph-classes-rules-and-pools.60013/

Thanks for you answer.

It is to extend the existing storage pool of, so adding a extra pool is not what i want.
Reweight the bigger disk at the cost of space it will be a better idee to just buy the cheaper 480 disks.

When replacing all the 480 gb disk for 960 all the disk will have more data and have a higher IO load.
I personally think the IO load is not going to slow down that much.

My biggest consure is the distributions of the PGs if all disk are at the same size (weight) it will get more even distributed over the disk.
Having different size disk in the same pool i think the distribution of de PGs will not be as even distributed.
End i don`t want to get in trouble later on because of that.

I will take it all in consideration.