Search results

Just a secondary thought, maybe try setting up a crush map that only includes the nodes in the one DC, create a pool with that map and see how it performs...

I would have expected them to run like lightning, all enterprise with PLP. Have you tried without spreading across the DC's? I believe the writes are synchronous and won't be acknowledged until writes to the replica OSDs have been acknowledged too, even if it's across to the secondary DC...

@hepo what hard drives is your Ceph cluster running?

We've started using Ceph ourselves and I have to say how good it is. The particular benefit is in the case of failover it'll pick up right where it left off, the cluster is self healing and VM migrations are wickedly fast as there's no disk to migrate. In our test environment I tried my best to...

From my testing of Proxmox, one frustration I had was that unlike my previous Xen environment, Proxmox does not detect if a VM has panicked/crashed/frozen and as such won't reboot the VM, potentially ending up in hours of downtime until the issue is realised and resolved. After a bit of digging...

The CephFS I was testing is mounted on the node itself. I've just given the writeback mode a go and it didn't make much difference. At least now I have some benchmarks to work to so when we get the new disks in I've got a good baseline. Thanks, Chris.

@mira thanks - that makes sense. So with the test run as you suggested. Directly against SSD I'm getting 20.0MiB/s with 5,366 IOPS. With CephFS I'm getting 420KiB/s with 105 IOPS (so a pretty significant drop). With SCSI mount I'm getting 144KiB/s with 36 IOPS (a huge drop! However much more...

Hi, config is below. Please note this is purely for testing Ceph performance, hence the multiple disks (scsi0, scsi1, virtio1) boot: order=scsi0;ide2;net0 cores: 4 ide2: cephfs:iso/ubuntu-20.04.3-live-server-amd64.iso,media=cdrom memory: 4096 meta: creation-qemu=6.1.0,ctime=1639144331 name...

Hi, I've been testing our Proxmox Ceph cluster and have noticed something interesting. I've been running fio benchmarks against a CephFS mount and within a VM using VirtIO SCSI. CephFS on /mnt/pve/cephfs - root@pve03:/mnt/pve/cephfs# fio --name=random-write --ioengine=posixaio --rw=randwrite...

@spirit do you think that moving the journals off to enterprise disks but keeping our actual storage on the Samsung EVO's will resolve our IOWait issue? I'd imagine it should because the fast disks will very quickly be able to tell Ceph where the data is. So from what you mention, our current...

Thanks for your contribution Spirit - perhaps back to the drawing board then! Chris.

So I've just run a test and from the VM I posted above. From monitoring the 'public' network and running an fio test I can see it's throwing data across the public network and distributing it to the other nodes. I wasn't expecting this as I'd assumed (perhaps wrongly?) that it would read...

No, EVO's fortunately :) Ideally would be good to have enterprise 12Gbps drives in there but they're hard to get hold of at the minute! We monitor the throughput constantly and haven't seen any real issues but from our tests although I can see we could get close to the mark so is certainly...

Hi, So I've been running some fio tests to see what the random read/write performance is like. I've got a Ceph pool called ceph_cluster (512 PGs) - this is the pool that all of the VM's sit on. I also have a CephFS mount on the PVE's called cephfs_data (32 PGs). The command I am running is -...

Hi Aaron, I made the changes with ratios and all has been cleverly recalculated and moved as needed. It's automatically chosen 512 placement groups for the main ceph_cluster pool. With regards to latency, each machine has 4 x 4TB 6Gbps Samsung 870 SSD's. There's one OSD to every disk and the...

Thanks Aaron - appreciate your help on this. I'll go ahead and change the scaling around a bit. With regards to making these changes, I presume it'll be a fairly intensive operation to refactor the 600GB or so we've got into new PG groups and we are likely to see some increased latency? Thanks...

Hi aaron, Thanks for your reply. In total I have four pools - device_health_metrics - 128 PGs / 128 Optimal PGs - 16MB used ceph-cluster - 64 PGs / 128 Optimal PGs - 499GB used (the one we'll be storing the majority of data ~7TB) cephfs_data - 128 PGs / 128 Optimal PGs - 4.6 GB (max of 500GB...

I've tried modifying the Ceph configuration to disable debugging and increase the number of threads for each OSD - # osd configuration osd_pool_default_min_size = 2 osd_pool_default_size = 3 osd_op_num_threads_per_shard_ssd = 4 # disable debugging debug ms=0 debug mds=0 debug osd=0 debug...

Hi, We're in the process of moving from a cluster that had a networked Dell ScaleIO storage to Proxmox using Ceph. Since moving across a couple of our VM's we've noticed quite an increase in iowait. Is this something that is typical of Ceph due to the nature of it's replication? For reference...

Hi oguz.... that would explain it! I assumed the GUI added them all. Sorry for the confusion and thanks for your help! Chris.