CEPH Speed drop off after a couple of GB transfered

[TomEnns]

I have a CEPH cluster with three nodes, the nodes are identical and configured as follows:

CPU: 15 3470 @ 3.20GHz
RAM: 16GB DDR3
Boot/Proxmox Disk: Samsung EVO 250GB SSD
WAL/DB Disk: Silicon Power 256GB SSD
OSD: Seagate 2TB HDD
NIC1: 1GbE used for proxmox management
NIC2: 1GbE assigned to CEPH Cluster only (Still through the same internal network, just separate IP and not in mesh mode)

I am running a VM with the latest LTS version of Ubuntu Server, on this server I have setup a Samba share that I am testing file transfers with to and from an SSD on a windows 11 PC.

The cluster seems to be running smoothly and I know I am not going to get crazy data throughput on 1GbE. However, I was hoping to get 80-100Mbps when reading or writing data to saturate the GbE connection. What I have found is that I will get these speeds for the first 0.5Gb to 1G transferred. After this point speed drops down to 30-50Mbps for the remainder of the transfer. I am having trouble understanding why it slows down after the first bit. Looking to see if there are settings I should be checking or something I should tweak. Or if this is to be expected with my setup?

I appreciate the insights in advance!

 

[bp2008]

Hmm, I am not an expert on ceph, having only tried it once in a nested virtualization scenario. I suspect you filled up a cache and started needing to wait for data to trickle over to the regular storage tier. https://docs.ceph.com/en/latest/rados/operations/cache-tiering/

But that only would explain why it slowed down after a time. Not why you could only sustain less than half a 1 Gbps transfer speed. For that I really have no idea.