Low Ceph Performance on 3-Node Proxmox 9 Cluster with SATA SSDs

P

pstojak

New Member
Aug 11, 2025
3
1
3
Hi all,

I'm running a 3-node Proxmox VE 9 cluster with Ceph and experiencing unexpectedly low performance during rados bench tests. I’d appreciate any insights or suggestions.

Cluster Setup:​

  • Proxmox VE 9 on all nodes
  • Ceph Quincy (default in PVE 9)
  • 3 nodes: pve01, pve02, pve03
  • 1 OSD per node, each on:
    • Samsung PM863a SATA SSD (480GB) — SAMSUNG_MZ7L3480HCHQ-00A07
  • Dedicated Ceph network using Proxmox SDN fabric
    • Nodes are connected in a ring topology via direct 10Gbps DAC
  • No separate WAL/DB devices
  • Ceph and public networks are separated

Benchmark Results:​

Using:
Code: 
rados bench -p testbench 60 write --no-cleanup

and
Code: 
rados bench -p testbench 60 rand

I get:

  • ~172 MB/s write
  • ~176 MB/s random read
  • ~42–43 IOPS
  • ~0.36–0.37s average latency

OSD Bench Results:​

Code: 
ceph tell osd.* bench
Each OSD reports ~525 MB/s throughput with 4MB blocks — so the disks themselves are capable.

iostat Output:​

During the benchmark:

  • sdb (OSD disk) shows only ~7–19% utilization
  • CPU is ~99% idle
  • Queue depth (aqu-sz) is low (~1.0–1.5)

What I’ve Tried:​

  • Checked SMART — all SSDs healthy
  • Ceph is clean, no recovery/backfill
Questions:

  1. Is this performance expected for a 3-node, 3-OSD Ceph cluster with SATA SSDs?
  2. Would adding more OSDs per node significantly improve performance?
  3. Would separating WAL/DB to NVMe help, even with SATA data devices?
  4. Any Ceph tuning recommendations for this setup?
  5. Could the SDN ring topology be a limiting factor?
Thanks in advance for any help or suggestions!
 
Does the Ceph network actually provide 10Gbit? Check with ethtool if they autonegiotated to the expected speed, and if, run some iperf tests between the nodes to see how much bandwidth it can provide.

Are both, Ceph Public and Ceph Cluster network on a 10Gbit Link or is one on a slower link?
 
  • Like
Reactions: Johannes S
Thanks for the quick reply!


  • Ceph Cluster Network: 10Gbps DAC, dedicated SDN fabric (ring topology)
  • Ceph Public Network: currently on 1Gbps over network switch

Code: 
root@pve01:~# iperf3 -c 192.168.88.4
Connecting to host 192.168.88.4, port 5201
[  5] local 192.168.88.3 port 38824 connected to 192.168.88.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec   42   1.32 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec    2   1.32 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.32 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.32 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.32 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.40 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.40 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.40 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.40 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.40 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec   45            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec                  receiver

iperf Done.
root@pve01:~# iperf3 -c 192.168.88.5
Connecting to host 192.168.88.5, port 5201
[  5] local 192.168.88.3 port 59018 connected to 192.168.88.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.16 GBytes  9.92 Gbits/sec    0   1.32 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.32 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.32 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.32 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.39 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.39 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.39 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.68 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.57 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   2.57 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec                  receiver

iperf Done.


Code: 
root@pve02:~# iperf3 -c 192.168.88.3
Connecting to host 192.168.88.3, port 5201
[  5] local 192.168.88.4 port 39166 connected to 192.168.88.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.86 Gbits/sec   75   1.52 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.52 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.91 Gbits/sec    4   1.52 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    5   1.52 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    3   1.52 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    5   1.52 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.52 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.52 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.52 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.53 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec   93            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.
root@pve02:~# iperf3 -c 192.168.88.5
Connecting to host 192.168.88.5, port 5201
[  5] local 192.168.88.4 port 38730 connected to 192.168.88.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.16 GBytes  9.92 Gbits/sec    0   1.46 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.91 Gbits/sec    0   1.46 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.46 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.46 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec                  receiver

iperf Done.
root@pve02:~#

Code: 
root@pve03:~# iperf3 -c 192.168.88.3
Connecting to host 192.168.88.3, port 5201
[  5] local 192.168.88.5 port 58488 connected to 192.168.88.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.89 Gbits/sec   90   1.37 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.88 Gbits/sec   12   1.28 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.30 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.88 Gbits/sec    0   1.30 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.88 Gbits/sec    1   1.30 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.88 Gbits/sec    1   1.30 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.30 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.88 Gbits/sec    0   1.30 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    1   1.30 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.88 Gbits/sec    0   1.30 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec  105            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.88 Gbits/sec                  receiver

iperf Done.
root@pve03:~# iperf3 -c 192.168.88.4
Connecting to host 192.168.88.4, port 5201
[  5] local 192.168.88.5 port 41974 connected to 192.168.88.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.16 GBytes  9.91 Gbits/sec   56   1.37 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.37 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.37 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.38 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.38 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.38 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.38 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.38 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.38 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.38 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec   59            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec                  receiver

iperf Done.
 
pstojak said:
Ceph Public Network: currently on 1Gbps over network switch
Click to expand...
not great, because the Ceph Cluster network is only used for the replication between the OSDs. Everything else, also IO from the guests to their virtual disks, is going via the Ceph Public network. Which is probably why you see the rather meh performance in the benchmark, as the benchmark client also uses the Ceph Public network to talk to the OSDs.

See the diagram here: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/

To change the Ceph Public network, you need to be a bit careful. Change the setting in the ceph.conf file. And then, stop, destroy and recreate each Ceph MON, one by one.
This way, they will be recreated with the new IP range and will update their internal monmap.

Other Ceph services, like OSDs, can be bulk restarted (per node!): systemctl restart ceph-osd.target.

You can use ss -tulpn | grep ceph before and throughout to make sure that the services listen on the new IPs.
 
aaron said:
not great, because the Ceph Cluster network is only used for the replication between the OSDs. Everything else, also IO from the guests to their virtual disks, is going via the Ceph Public network. Which is probably why you see the rather meh performance in the benchmark, as the benchmark client also uses the Ceph Public network to talk to the OSDs.

See the diagram here: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/

To change the Ceph Public network, you need to be a bit careful. Change the setting in the ceph.conf file. And then, stop, destroy and recreate each Ceph MON, one by one.
This way, they will be recreated with the new IP range and will update their internal monmap.

Other Ceph services, like OSDs, can be bulk restarted (per node!): systemctl restart ceph-osd.target.

You can use ss -tulpn | grep ceph before and throughout to make sure that the services listen on the new IPs.
Click to expand...
Thanks!

I’ve now moved the Ceph public network to the same 10Gbps SDN fabric as the Ceph cluster network.


Screenshot 2025-08-21 145529.jpg
​

 
Last edited:
  • Like
Reactions: UdoB
Looks a lot better :)
 
You must log in or register to reply here.
Top Bottom