Slow ceph operation

[kacper.adrianowicz]

Hi

I have 3 node cluster with 2 x Kioxia 1,92TB SAS Enterprise SSD. Disk operation on VM's are very slow. Network config is as follow:

Each node have:
1 x 1Gb NIC for Ceph public network (the same network as proxmox itself)
1 x 10Gb NIC for Ceph cluster network

I'm no expert in ceph and couldn't figure out what is the issue. I read that problem might be that i have 1Gb NIC for public ceph network that is also used by proxmox.

Is this assumption correct? Should I have seperate 10Gb NIC for ceph public network? Does it have to be separate from ceph cluster network or can it be the same NIC?

Below Ceph config

[global]
    auth_client_required = cephx
    auth_cluster_required = cephx
    auth_service_required = cephx
    cluster_network = 10.99.99.10/27
    fsid = 6eb06b21-c9f2-4527-8675-5dc65872dde9
    mon_allow_pool_delete = true
    mon_host = 10.0.5.10 10.0.5.11 10.0.5.12
    ms_bind_ipv4 = true
    ms_bind_ipv6 = false
    osd_pool_default_min_size = 2
    osd_pool_default_size = 3
    public_network = 10.0.5.10/27

[client]
    keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
    keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.g-srv-01]
    public_addr = 10.0.5.10

[mon.g-srv-02]
    public_addr = 10.0.5.11

[mon.g-srv-03]
    public_addr = 10.0.5.12

 

[VictorSTS]

Ceph Public is the network used to read/write from/to your Ceph OSDs from each PVE host, so your are limited to 1GB/s. Ceph Cluster network is used for OSD replication traffic only. Move Ceph Public to your 10GB nic and there should be an improvement. You can share the same nic for both Ceph Public and Ceph Cluster.

 

[aaron]

1 x 1Gb NIC for Ceph public network (the same network as proxmox itself)

Move that to a fast network too! Otherwise your guests will be limited by that 1 Gbit. see https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/ for which ceph network is used for what.
Checkout our 2023 Ceph benchmark whitepaper (in the sticky threads). A 10 Gbit network will quickly become the bottleneck if you use fast datacenter NVMEs.

Moving a Ceph network can be done on the fly, but especially for the Ceph Public network, the procedure is a little bit more involved.
How you can change the Ceph network config for a running cluster is explained here: https://lore.proxmox.com/all/20260102165754.650450-1-a.lauterer@proxmox.com/

It is the patch for our documentation. Once a new version of the admin guide is created, it will be there in a more readable form. But that patch should still be readable enough

 

[VictorSTS]

Moving a Ceph network can be done on the fly, but especially for the Ceph Public network, the procedure is a little bit more involved.

Nice to see this reaching the official documentation!

Maybe OP did setup a vlan for Ceph Public network with different IP network from that of other cluster services and can just move the vlan to a different physical nic/bond. Did you @kacper.adrianowicz ? If you didn't, you could do it now