Proxmox Ceph

flashback830 · Aug 9, 2025

Hello everyone,

I’m getting a bit desperate at this point. I have 3 Proxmox servers, two of which are supposed to run Ceph. Ceph itself works, but apparently not over the intended 10 G connection — it’s only using a 1 G connection. The third server is basically just acting as an additional monitor/manager.
In the two Ceph servers, there are 6 SSDs each (OSD) plus 2 SSDs in a RAID 1 for the OS.

My problem is this: ( The picture is from a VM )

Obviously, that’s far from what SSDs and a 10 G link should be delivering.

The configuration is as follows:
192.168.25.0/24 – for the VMs themselves (1 G)
192.168.26.0/24 – management VLAN (1 G)
192.168.30.0/24 – transfer for Ceph (10 G)

Server 1:

Code:

root@DC3-R630:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:9c brd ff:ff:ff:ff:ff:ff
    altname enp6s0f0
    altname enx246e96748d9c
3: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:9d brd ff:ff:ff:ff:ff:ff
    altname enp6s0f1
    altname enx246e96748d9d
4: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master trans1 state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:98 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
    altname enx246e96748d98
5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 24:6e:96:74:8d:9a brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
    altname enx246e96748d9a
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:9d brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.240/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe74:8d9d/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
7: trans1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.240/24 scope global trans1
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe74:8d98/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:6e:96:74:8d:9c brd ff:ff:ff:ff:ff:ff
    inet 192.168.25.240/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe74:8d9c/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
9: veth504i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:30:9a:07:6c:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
14: tap1000i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 76:96:3f:34:d7:e1 brd ff:ff:ff:ff:ff:ff
15: tap500i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 36:26:76:d2:04:1c brd ff:ff:ff:ff:ff:ff
16: veth502i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:c9:a8:7e:f0:79 brd ff:ff:ff:ff:ff:ff link-netnsid 1
17: veth503i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:6b:df:33:47:65 brd ff:ff:ff:ff:ff:ff link-netnsid 2
18: veth501i0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether fe:7a:41:fa:01:58 brd ff:ff:ff:ff:ff:ff link-netnsid 3

Code:

root@DC3-R630:~# iperf3 -c 192.168.30.241
Connecting to host 192.168.30.241, port 5201
[  5] local 192.168.30.240 port 46976 connected to 192.168.30.241 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.91 Gbits/sec   29   1.56 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.88 Gbits/sec    2   1.58 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.60 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.62 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.72 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.89 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   2.18 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.88 Gbits/sec    0   2.53 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   2.53 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   2.53 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec   31            sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.

Code:

[global]
cluster_network = 192.168.30.0/24
    auth_client_required = cephx
    auth_cluster_required = cephx
    auth_service_required = cephx
    fsid = 2c73b9cd-e0d5-42a6-a47b-b713ac77000f
    mon_allow_pool_delete = true
    mon_host = 192.168.26.240 192.168.26.241 192.168.26.230
    ms_bind_ipv4 = true
    ms_bind_ipv6 = false
    osd_pool_default_min_size = 2
    osd_pool_default_size = 3
    public_network = 192.168.26.0/24

[client]
    auth_client_required = cephx
    auth_cluster_required = cephx
    auth_service_required = cephx
    keyring = /etc/pve/priv/$cluster.$name.keyring
    mon_allow_pool_delete = true
    ms_bind_ipv6 = false
    osd_pool_default_min_size = 1
    osd_pool_default_size = 2
    public_network = 192.168.26.0/24

[client.crash]
    keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.DC3-R630]
    public_addr = 192.168.26.240

[mon.DC4-R630]
    public_addr = 192.168.26.241

mon_host = 192.168.26.240,192.168.26.241,192.168.26.230
mon_initial_members = DC3-R630, DC4-R630, PLEX

Server 2:

Code:

root@DC4-R630:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 24:6e:96:12:4d:4c brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0
    altname enx246e96124d4c
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 24:6e:96:12:4d:4d brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
    altname enx246e96124d4d
4: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master trans1 state UP group default qlen 1000
    link/ether a0:36:9f:e0:79:78 brd ff:ff:ff:ff:ff:ff
    altname enxa0369fe07978
5: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether 24:6e:96:12:4d:4e brd ff:ff:ff:ff:ff:ff
    altname enp1s0f2
    altname enx246e96124d4e
6: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 24:6e:96:12:4d:4f brd ff:ff:ff:ff:ff:ff
    altname enp1s0f3
    altname enx246e96124d4f
7: enp4s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a0:36:9f:e0:79:7a brd ff:ff:ff:ff:ff:ff
    altname enxa0369fe0797a
8: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f4:8e:38:cd:4d:83 brd ff:ff:ff:ff:ff:ff
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:6e:96:12:4d:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.241/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe12:4d4f/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:6e:96:12:4d:4e brd ff:ff:ff:ff:ff:ff
    inet 192.168.25.241/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:fe12:4d4e/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
11: trans1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether a0:36:9f:e0:79:78 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.241/24 scope global trans1
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:fee0:7978/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
13: tap2000i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vmbr1 state UNKNOWN group default qlen 1000
    link/ether e6:ed:cc:95:a1:1d brd ff:ff:ff:ff:ff:ff

Code:

root@DC4-R630:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.30.240, port 46964
[  5] local 192.168.30.241 port 5201 connected to 192.168.30.240 port 46976
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec
[  5]  10.00-10.00  sec  1.25 MBytes  8.98 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

Code:

[global]
cluster_network = 192.168.30.0/24
    auth_client_required = cephx
    auth_cluster_required = cephx
    auth_service_required = cephx
    fsid = 2c73b9cd-e0d5-42a6-a47b-b713ac77000f
    mon_allow_pool_delete = true
    mon_host = 192.168.26.240 192.168.26.241 192.168.26.230
    ms_bind_ipv4 = true
    ms_bind_ipv6 = false
    osd_pool_default_min_size = 2
    osd_pool_default_size = 3
    public_network = 192.168.26.0/24

[client]
    auth_client_required = cephx
    auth_cluster_required = cephx
    auth_service_required = cephx
    keyring = /etc/pve/priv/$cluster.$name.keyring
    mon_allow_pool_delete = true
    ms_bind_ipv6 = false
    osd_pool_default_min_size = 1
    osd_pool_default_size = 2
    public_network = 192.168.26.0/24

[client.crash]
    keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.DC3-R630]
    public_addr = 192.168.26.240

[mon.DC4-R630]
    public_addr = 192.168.26.241

mon_host = 192.168.26.240,192.168.26.241,192.168.26.230
mon_initial_members = DC3-R630, DC4-R630, PLEX

I hope someone can help me out

UdoB · Aug 9, 2025

flashback830 said:
I have 3 Proxmox servers, two of which are supposed to run Ceph.

Make sure to understand the limitations of two Ceph nodes.

Personally I came to the conclusion that I wanted at least five! https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

Disclaimer: not using Ceph currently...

SteveITS · Aug 9, 2025

Are these enterprise SSDs?

public_network Is where the initial disk write happens. The Ceph private net is where the replication happens (the copy, rebalance, etc.).

mfed · Aug 14, 2025

public_network is what is used by the Ceph clients to connect to the Ceph data. I believe you need to move your public_network to your 10Gb/s subnet

Johannes S · Aug 14, 2025

Please check that your setup match the recommendations at https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster and https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network
A two-node Ceph cluster is really not what you should run in a production environment so you will need at least three nodes for hosting the stored data.

Regarding performance from the first linked wiki article:

If unsure, we recommend using three (physical) separate networks for high-performance setups:

one very high bandwidth (25+ Gbps) network for Ceph (internal) cluster traffic.

one high bandwidth (10+ Gpbs) network for Ceph (public) traffic between the ceph server and ceph client storage traffic. Depending on your needs this can also be used to host the virtual guest traffic and the VM live-migration traffic.

one medium bandwidth (1 Gbps) exclusive for the latency sensitive corosync cluster communication.

https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster

Search

Search

Proxmox Ceph

flashback830

New Member

UdoB

Distinguished Member

SteveITS

Active Member

mfed

Well-Known Member

Johannes S

Famous Member

We value your privacy