Ok I require some guidance here. My concern is overwriting files or damaging the file system whenver someone mentions fio or rados benchmarking. Is it safe to run a rados or fio benchmark while an OSD is within an active (production) CEPH pool? Or should I remove it from my pool and create a secondary pool for benchmarking?
To benchmark a single disk, you will have to take it out of the cluster as a direct FIO benchmark will be destructive. Meaning, marking the OSD as out, waiting for the cluster to become healthy again and then destroying the OSD. The Rados benchmark needs to be done with a working cluster and pool. Ideally you can do that in some off time when you can power off the VMs to not have any load on the cluster other than the benchmark. If that is not possible, at least a time when the load of the benchmark will not have too much of a negative effect on the VMs. And let it run for a while to reduce the impact of short term load spikes or any caches. We usually let our benchmarks run for 10 minutes.
Thank you for linking that benchmark paper, yes that's very clearly explained on pages 5 and 6, it gave some insight into how CEPH works but I didn't see "why" that was the case, for example was there a network bottleneck or is it a latency issue with CEPH that causes a single client to never saturate all of the available bandwidth?
If the network had been the bottleneck, then the overall performance would not have increased with multiple benchmarks at the same time. No, it is more that a single client itself is not fast enough to saturate the whole cluster, running itself into some local CPU bound problems, maybe also some latency.
I may need some more clarification, but this is how the majority of my VMs are configured:
Okay, so the disk bus type is set to IDE which is primarily there for compatibility reasons. Ideally you would switch that to SCSI. In combination with the VirtIO SCSI controller (already set) you should see better disk IO because the VirtIO implementation is a much thinner layer than emulating a full IDE device (or SATA for that matter).
To switch a Windows VM to VirtIO SCSI you need to do a few steps, unfortunately. They are described in our
Proxmox VE Wiki (last section). The VirtIO driver ISO for Windows can be
downloaded from the Fedora project.
In our experience, it is very important to first attach a small dummy disk with the SCSI bus to the running VM before you shut it down and change the actual disk to SCSI. Otherwise Windows will not be able to boot from a SCSI disk.
To change the bus type of the existing disk, you need to detach it. It will show up as unused disk. You can then edit it to attach it again. Make shure to set the Bus Type to SCSI. Also don't forget to adjust the boot order in the VMs Options afterwards. This is something that I like to forget.
You mean there is a way to prevent a CEPH OSD from using the same CPU thread as the VM?? Please explain.
No, usually, the Disk IO of a VM is handled in the same thread as the VM itself. IO Threading moves the handling of the Disk IO into a separate thread. This has nothing to do with Ceph but only the virtualization (KVM/Qemu).
There is no target size set, mostly because I'm concerned that I will set a target that is too large. And to answer your other question, the 256 optimal came from "Datacentre>Node>Ceph>Pool" if you look there, the pool will have columns for "# of PGs" and "Optimal # of PGs".
That number comes from the Ceph autoscaler. If there is no target_size or target_ratio defined, it will only take the current pool size into account. But this means, that if the pool grows it could happen that the autoscaler decides that a new, larger pg_num is better and you will have to do some rebalancing which could affect the performance of the cluster. If you have somewhat of an idea how much of the whole cluster will be used by the pool, you can tell that to the autoscaler by setting either the target_size or target_ratio parameter. It can then determine the best pg_num right away, saving you a rebalance later on.
The target_ratio takes precedence if you define both, and is in my opinion also the better parameter. If you only have one pool, then it is easy, set the target_ratio to 1 as it is the only pool.
You should also be aware that the autoscaler only becomes active if the better pg_num is off from the current one by a factor of 3. So if the ideal pg_num is 512 and the current one is 256, the autoscaler should not become active, as it is off by a factor of 2, but you can still set it manually.
You only changed the pg_num?
