Search results

Hi Aaron, I've had a breakthrough on this today. I have 3 nodes that were originally spec'd by our provider with Samsung EVO 870's (useless for Ceph) so we're in the process of swapping them out with Samsung PM893's. We added one PM893 to each node just to test and this is where the benchmarks...

Hi, We've installed a 3 node Proxmox cluster running Ceph. On the cluster we have two VM's. When we reboot all PVE nodes, the IOWait of the VM's drops to pretty much nothing. However over time, the IOWait creeps up - there is no load on these VM's. Any idea why this might be? As you can see...

@spirit those are pretty good stats. We've just replaced our cluster (3 nodes, 1 DC) with 3 x 3.84TB Samsung P893's (with another 9 on order, so 4 per node). We have Intel Xeon E5-2697 configured in performance mode so 2.3GHz / 3.6GHz turbo per core. Running the same FIO benchmark within a VM...

@hepo have you got any further with this? Have you tried running a rados benchmark against a newly created pool? We're also finding writes are heavily restricted when running on a VM itself.

Just a secondary thought, maybe try setting up a crush map that only includes the nodes in the one DC, create a pool with that map and see how it performs...

I would have expected them to run like lightning, all enterprise with PLP. Have you tried without spreading across the DC's? I believe the writes are synchronous and won't be acknowledged until writes to the replica OSDs have been acknowledged too, even if it's across to the secondary DC...

@hepo what hard drives is your Ceph cluster running?

We've started using Ceph ourselves and I have to say how good it is. The particular benefit is in the case of failover it'll pick up right where it left off, the cluster is self healing and VM migrations are wickedly fast as there's no disk to migrate. In our test environment I tried my best to...

From my testing of Proxmox, one frustration I had was that unlike my previous Xen environment, Proxmox does not detect if a VM has panicked/crashed/frozen and as such won't reboot the VM, potentially ending up in hours of downtime until the issue is realised and resolved. After a bit of digging...

The CephFS I was testing is mounted on the node itself. I've just given the writeback mode a go and it didn't make much difference. At least now I have some benchmarks to work to so when we get the new disks in I've got a good baseline. Thanks, Chris.

@mira thanks - that makes sense. So with the test run as you suggested. Directly against SSD I'm getting 20.0MiB/s with 5,366 IOPS. With CephFS I'm getting 420KiB/s with 105 IOPS (so a pretty significant drop). With SCSI mount I'm getting 144KiB/s with 36 IOPS (a huge drop! However much more...

Hi, config is below. Please note this is purely for testing Ceph performance, hence the multiple disks (scsi0, scsi1, virtio1) boot: order=scsi0;ide2;net0 cores: 4 ide2: cephfs:iso/ubuntu-20.04.3-live-server-amd64.iso,media=cdrom memory: 4096 meta: creation-qemu=6.1.0,ctime=1639144331 name...

Hi, I've been testing our Proxmox Ceph cluster and have noticed something interesting. I've been running fio benchmarks against a CephFS mount and within a VM using VirtIO SCSI. CephFS on /mnt/pve/cephfs - root@pve03:/mnt/pve/cephfs# fio --name=random-write --ioengine=posixaio --rw=randwrite...

@spirit do you think that moving the journals off to enterprise disks but keeping our actual storage on the Samsung EVO's will resolve our IOWait issue? I'd imagine it should because the fast disks will very quickly be able to tell Ceph where the data is. So from what you mention, our current...

Thanks for your contribution Spirit - perhaps back to the drawing board then! Chris.

So I've just run a test and from the VM I posted above. From monitoring the 'public' network and running an fio test I can see it's throwing data across the public network and distributing it to the other nodes. I wasn't expecting this as I'd assumed (perhaps wrongly?) that it would read...

No, EVO's fortunately :) Ideally would be good to have enterprise 12Gbps drives in there but they're hard to get hold of at the minute! We monitor the throughput constantly and haven't seen any real issues but from our tests although I can see we could get close to the mark so is certainly...

Hi, So I've been running some fio tests to see what the random read/write performance is like. I've got a Ceph pool called ceph_cluster (512 PGs) - this is the pool that all of the VM's sit on. I also have a CephFS mount on the PVE's called cephfs_data (32 PGs). The command I am running is -...

Hi Aaron, I made the changes with ratios and all has been cleverly recalculated and moved as needed. It's automatically chosen 512 placement groups for the main ceph_cluster pool. With regards to latency, each machine has 4 x 4TB 6Gbps Samsung 870 SSD's. There's one OSD to every disk and the...

Thanks Aaron - appreciate your help on this. I'll go ahead and change the scaling around a bit. With regards to making these changes, I presume it'll be a fairly intensive operation to refactor the 600GB or so we've got into new PG groups and we are likely to see some increased latency? Thanks...