fstrim performance on CEPH RBD with Proxmox

[Leah]

Hey,

we observe major performance issues while running fstrim on VMs backed by a SSD pool (3 replica, 50OSDs) with Ceph (16.2.7) on proxmox. We have a workload that leads to some bigger data fluctuation on our VMs (CentOS 7) . Therefore we have enabled the discard mode for the disks and run fstrim once a week. First we observed major performance problems if multiple fstrim processes where running at the same time. To mitigate the first problem we wrote some software that scheduled the fstrim runs in a way that we can make sure only one is running at a time on the whole cluster. This made the situation somewhat better but didn't solve the problem. We still observe near blocking of the IO performance while the fstrim runs and we are not sure why. Today we noticed that some strange reported outages by our monitoring are also caused by the fstrim runs, the node exporter needs 135seconds to read 40KB from disk at these moments. Thats totally not what we expected. Is there a known issue with fstrim on rbd devices containing a XFS file system? Or at general are there parameters to tune to get a better performance for small reads and writes?

 

[aaron]

What kind of OSDs do you have that cluster?
It is possible that deleting objects might eat a lot of performance. You could try to increase the sleep time between delete operations and observe if the impact is less.

ceph config set osd osd_delete_sleep 2

See https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#miscellaneous

You can also check if you have some OSDs that underperform.
To so, you can run ceph tell osd.* bench and compare the outputs for some outliers.

 

[Leah]

The OSDs are Intel SSD D3-S4510 3.84TB SSDs. Will take a look into the other parameters tomorrow.

 

[guletz]

Hi,

I think you could try to run fstrim more often that 1/week. During one week you will need to send more I/O to your storage (whatever it is) compared with a let say a daily/short period. Like ... hey Mr. Storage, please take care of this 300k blocks that are not in use, compared with 2k blocks.

Another factor could be the total size of your VM virtDisk. It is a difference if for example you have a total of 100k blocks in a vDisk but you use 10k and is other story if you have let say 20k blocks in the same vDisk with the SAME 10k blocks of data .

Many blocks to read many I/O to the storage.


Block size of your VM .... could be not the same of your storage so -> phisical block size of the underlying HDd/ SSD

This is my own understanding about fstrim, I may be wrong but this is what I see ....

Good luck/Bafta !

 

[Leah]

You can also check if you have some OSDs that underperform.
To so, you can run ceph tell osd.* bench and compare the outputs for some outliers.

We found another issue with this command, so thanks! But unfortunatly we have the problem on all our SSD Pools and not only on the affected one. So for the fstrim problem it looks like everything is performing as expected.

ceph config set osd osd_delete_sleep 2

Applied it and now we will see if something changes.

I think you could try to run fstrim more often that 1/week.

We will try this as one of the next things but first tests don't look promising. It nearly needs the same time and performance like running it once a week even if we run it two times or more in a row.

Another factor could be the total size of your VM virtDisk. It is a difference if for example you have a total of 100k blocks in a vDisk but you use 10k and is other story if you have let say 20k blocks in the same vDisk with the SAME 10k blocks of data .

Our disks usually have 600GB with 60%-85% used. So this shouldn't apply here.

Block size of your VM .... could be not the same of your storage so

Need to have a look at that.