[SOLVED] Proxmox ceph network configuration

[able]

Hi,
I read this sentence on ceph hardware-recommendations : "Provision at least 10 Gb/s networking in your datacenter, both among Ceph hosts and between clients and your Ceph cluster"
This is my ceph configuration:

[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 192.168.4.34/24
     fsid = 9d805c38-f944-4e24-8e6c-6b278011c85f
     mon_allow_pool_delete = true
     mon_host = 192.168.3.34 192.168.3.35 192.168.3.36
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 192.168.3.34/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.1-34-0001]
     public_addr = 192.168.3.34

[mon.1-35-0001]
     public_addr = 192.168.3.35

[mon.1-36-0001]
     public_addr = 192.168.3.36

My proxmox server "1-34-0001" /etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno4 inet manual

iface eno3 inet manual

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.34/24
    gateway 192.168.1.1
    bridge-ports eno4
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet static
    address 192.168.3.34/24
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0

auto vmbr2
iface vmbr2 inet static
    address 192.168.4.34/24
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

My virtual machine uses vmbr0. vmbr1 as the public network of ceph, vmbr2 as the cluster network of ceph. However, the 10 GB NIC only has eno1 and eno2.

Please give me some advice on the following questions:
1. Whether the vm server uses a 1GB NIC to communicate with the ceph public network seriously affects the performance.
2. If the current configuration performance is poor, can the cluster network use vmbr1 with the public network and vm server use vmbr2 ?

 

[able]

Provision at least 10 Gb/s networking in your datacenter, both among Ceph hosts and between clients and your Ceph cluster

I suspect I misunderstand the term "client". In proxmox ceph cluster, does "client" refer to proxmox server (1-34-0001) or vm server?

 

[gfngfn256]

The docs you quote are Ceph's official documentation - not Proxmox. Client's referred to in those docs are those accessing Ceph storage, not hosted locally on the client.

 

[able]

The docs you quote are Ceph's official documentation - not Proxmox. Client's referred to in those docs are those accessing Ceph storage, not hosted locally on the client.

Thanks a lot for your reply.
Proxmox documentation Initial Ceph Installation & Configuration mentions the "Separating your Ceph traffic".
Does this mean that a healthy configuration in proxmox ceph cluster is to have the ceph public network use a separate 10 GB NIC and the vm traffic is separate from it?

 

[gfngfn256]

Proxmox documentation Initial Ceph Installation & Configuration mentions the "Separating your Ceph traffic"

I believe this is referring to Ceph cluster vs Proxmox cluster/corosync. So in short, its got nothing to do with VM traffic.
Please note: I don't personally use Ceph.

 

[able]

I believe this is referring to Ceph cluster vs Proxmox cluster/corosync. So in short, its got nothing to do with VM traffic.
Please note: I don't personally use Ceph.

Thank you for your answers, I think they are in line with the basic philosophy.

 

[B.Otto]

Thanks a lot for your reply.
Proxmox documentation Initial Ceph Installation & Configuration mentions the "Separating your Ceph traffic".
Does this mean that a healthy configuration in proxmox ceph cluster is to have the ceph public network use a separate 10 GB NIC and the vm traffic is separate from it?

Yeah, best practise is to separate your CEPH network from your other network, since CEPH creates a lot of traffic, and a slow Storage network might make everything slow.

• CEPH + Proxmox Cluster network (Corosync) on the same link is not good because Corosync expects low latency and high CEPH traffic might cause Corosync to think that the cluster is not healthy
• CEPH + VM Network on the same link might slow or disrupt your VM networking

There are options for both Ceph Public and Ceph Cluster Network (which is NOT Proxmox Cluster network), but the second one is optional. If there is no dedicated Ceph Cluster network assigned, it will also use the Ceph Public link. In a small cluster you could go with Ceph Public only.