Proxmox 9.2.2 with Ceph Squid bad write performance

[Maurice Janssen]

We have build a 3 node cluster with 6 SSD in each node configure for ceph. After we have finish the configuration the read performance was very good (2 GB/s with low latency) but the write performance is very bad (200MB/s with a very high latency). I have checked everything but can`t find the issue. Network is a 20Gbit Bond (2*10Gbit) for ceph traffic with jumbo frames enabled. And there are no errors in the logs. Any idea somebody ?

 

[complexplaster27]

What are you using to test the performance? Any tools? Settings? Applications? Inside a VM?

 

[SteveITS]

What drive models? Enterprise/PLP?

 

[Maurice Janssen]

we are testing with windows host with diskspd , we are using enterprice

Intel D5-P4320 7.68TB disks​

 

[daanw]

Read (sequential) 3.200 MB/s
Write (sequential) 1.000 MB/s
Read (random 4K, QD32) 427.000 IOPS
Write (random 4K, QD32) 36.000 IOPS

This SSD seems geared towards read performance by design.

 

[spirit]

We have build a 3 node cluster with 6 SSD in each node configure for ceph. After we have finish the configuration the read performance was very good (2 GB/s with low latency) but the write performance is very bad (200MB/s with a very high latency). I have checked everything but can`t find the issue. Network is a 20Gbit Bond (2*10Gbit) for ceph traffic with jumbo frames enabled. And there are no errors in the logs. Any idea somebody ?

200MB/S with which block size ? iodepth ?

 

[SteveITS]

There are tests for Ceph that can be run on the host and avoid any potential VM issues. For a Windows VM, ensure you're following the guide to use SCSI, as well as not using a "host" CPU type (see various forum threads).