Hi everyone,
I have a three node Proxmox VE Cluster to test Ceph for our enviroment.
Specs:
which seems to be ok for me
When I setup a vm on Proxmox using the RBD Storage, I tried to benchtest the read/write performance of the VM with fio:
The write performance seems very slow to me.
Also tested the blockstorage with fio from my promox server:
I also created a second rbd pool and ran the fio test on both pools simultaneously and both ended up with the same performance as shown before (not more that 127 Mb/s).
For me it seems that the speed is capped somehow.
Ceph Config:
I am new to ceph and proxmox and this is for my final project, so please sorry any dumb statements i might written.
Thank you
Tobi
I have a three node Proxmox VE Cluster to test Ceph for our enviroment.
Specs:
- ASUS Server KNPA-U16 Series
- 32 x AMD EPYC 7281 16-Core Processor (1 Socket)
- 64GB RAM
- two 1GB Nics (one used for to access the proxmox web GUI)
- two 10GB Nics used for ceph-cluster (hosts connceted over a 10GB switch and tested the speed with iperf)
- Network is seperated into Public Network and Cluster Network
- Used in Bond mode access the Networks via VLAN (LACP is used)
- 4x 1TB Samsung SSD (SAMSUNG_MZ7KM960HAHP-00005) directly plugged into the ASUS Chassis
Code:
rados -p ceph_rbd2 bench 10 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ce-cl-101_613665
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 278 262 1047.92 1048 0.0634726 0.0588222
2 16 575 559 1117.89 1188 0.040781 0.0565382
3 16 847 831 1107.84 1088 0.0399402 0.0574625
4 16 1126 1110 1109.74 1116 0.0372452 0.0569344
5 16 1408 1392 1113.36 1128 0.0680987 0.057054
6 16 1699 1683 1121.78 1164 0.0855339 0.0568148
7 16 1989 1973 1127.22 1160 0.0954367 0.0565672
8 16 2271 2255 1127.3 1128 0.0777163 0.0565807
9 16 2572 2556 1135.8 1204 0.0575054 0.0562017
10 16 2863 2847 1138.61 1164 0.0351853 0.0559141
Total time run: 10.0422
Total writes made: 2863
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1140.39
Stddev Bandwidth: 47.1046
Max bandwidth (MB/sec): 1204
Min bandwidth (MB/sec): 1048
Average IOPS: 285
Stddev IOPS: 11.7762
Max IOPS: 301
Min IOPS: 262
Average Latency(s): 0.0559936
Stddev Latency(s): 0.0240188
Max latency(s): 0.288094
Min latency(s): 0.0220342
Cleaning up (deleting benchmark objects)
Removed 2863 objectswhich seems to be ok for me
When I setup a vm on Proxmox using the RBD Storage, I tried to benchtest the read/write performance of the VM with fio:
Code:
fio --rw=write --name=test --size=5G --direct=1 --bs=1M
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=119MiB/s][w=119 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2317: Thu Oct 14 09:25:18 2021
write: IOPS=117, BW=117MiB/s (123MB/s)(5120MiB/43645msec); 0 zone resets
clat (usec): min=5998, max=19205, avg=8460.79, stdev=1176.34
lat (usec): min=6072, max=19244, avg=8520.29, stdev=1176.09
clat percentiles (usec):
| 1.00th=[ 6390], 5.00th=[ 6718], 10.00th=[ 6980], 20.00th=[ 7439],
| 30.00th=[ 7832], 40.00th=[ 8094], 50.00th=[ 8356], 60.00th=[ 8717],
| 70.00th=[ 8979], 80.00th=[ 9372], 90.00th=[10028], 95.00th=[10552],
| 99.00th=[11731], 99.50th=[12256], 99.90th=[13829], 99.95th=[14353],
| 99.99th=[19268]
bw ( KiB/s): min=112640, max=129024, per=99.97%, avg=120091.36, stdev=3620.92, samples=87
iops : min= 110, max= 126, avg=117.24, stdev= 3.56, samples=87
lat (msec) : 10=90.02%, 20=9.98%
cpu : usr=0.91%, sys=0.75%, ctx=5206, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=5120MiB (5369MB), run=43645-43645msec
Disk stats (read/write):
dm-0: ios=0/5492, merge=0/0, ticks=0/46052, in_queue=46052, util=99.77%, aggrios=0/5348, aggrmerge=0/144, aggrticks=0/45036, aggrin_queue=34592, aggrutil=99.77%
sda: ios=0/5348, merge=0/144, ticks=0/45036, in_queue=34592, util=99.77%
Code:
fio --rw=read --name=test --size=5G --direct=1 --bs=1M
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=224MiB/s][r=224 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2321: Thu Oct 14 09:26:26 2021
read: IOPS=215, BW=216MiB/s (226MB/s)(5120MiB/23711msec)
clat (msec): min=3, max=215, avg= 4.63, stdev= 3.00
lat (msec): min=3, max=215, avg= 4.63, stdev= 3.00
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 6], 95.00th=[ 6],
| 99.00th=[ 7], 99.50th=[ 7], 99.90th=[ 9], 99.95th=[ 14],
| 99.99th=[ 215]
bw ( KiB/s): min=131072, max=233472, per=100.00%, avg=221358.30, stdev=15084.94, samples=47
iops : min= 128, max= 228, avg=216.17, stdev=14.73, samples=47
lat (msec) : 4=14.94%, 10=84.96%, 20=0.06%, 50=0.02%, 250=0.02%
cpu : usr=0.24%, sys=1.64%, ctx=5155, majf=0, minf=268
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=216MiB/s (226MB/s), 216MiB/s-216MiB/s (226MB/s-226MB/s), io=5120MiB (5369MB), run=23711-23711msec
Disk stats (read/write):
dm-0: ios=5110/4, merge=0/0, ticks=23192/12, in_queue=23204, util=99.60%, aggrios=5120/3, aggrmerge=0/1, aggrticks=23376/9, aggrin_queue=17064, aggrutil=99.56%
sda: ios=5120/3, merge=0/1, ticks=23376/9, in_queue=17064, util=99.56%The write performance seems very slow to me.
Also tested the blockstorage with fio from my promox server:
Code:
fio -ioengine=rbd -name=test -bs=1M -iodepth=1 -rw=randwrite -runtime=60 -pool=ceph_rbd -rbdname=test
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=rbd, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=122MiB/s][w=122 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=616821: Thu Oct 14 11:30:21 2021
write: IOPS=117, BW=117MiB/s (123MB/s)(7046MiB/60005msec); 0 zone resets
slat (usec): min=88, max=723, avg=154.15, stdev=34.38
clat (msec): min=5, max=132, avg= 8.36, stdev= 1.78
lat (msec): min=6, max=132, avg= 8.51, stdev= 1.78
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8],
| 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9],
| 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 10], 95.00th=[ 10],
| 99.00th=[ 11], 99.50th=[ 12], 99.90th=[ 13], 99.95th=[ 14],
| 99.99th=[ 133]
bw ( KiB/s): min=88064, max=126976, per=100.00%, avg=120298.49, stdev=4065.18, samples=119
iops : min= 86, max= 124, avg=117.48, stdev= 3.97, samples=119
lat (msec) : 10=95.98%, 20=4.00%, 250=0.01%
cpu : usr=2.05%, sys=0.14%, ctx=7064, majf=0, minf=1822
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,7046,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=7046MiB (7388MB), run=60005-60005msec
Disk stats (read/write):
dm-1: ios=0/1497, merge=0/0, ticks=0/568, in_queue=568, util=1.49%, aggrios=24/1040, aggrmerge=0/586, aggrticks=8/3350, aggrin_queue=3419, aggrutil=1.67%
nvme0n1: ios=24/1040, merge=0/586, ticks=8/3350, in_queue=3419, util=1.67%I also created a second rbd pool and ran the fio test on both pools simultaneously and both ended up with the same performance as shown before (not more that 127 Mb/s).
For me it seems that the speed is capped somehow.
Ceph Config:
Code:
global]
mon_max_pg_per_osd = 400
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.8.16.1/16
fsid = e9e5a603-7fc1-4ea7-9f84-04a632079588
mon_allow_pool_delete = true
mon_host = 10.6.14.1 10.6.14.2 10.6.14.3
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.6.14.1/16
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[mds.ce-cl-101]
host = ce-cl-101
mds standby for name = pve
[mon.ce-cl-101]
public_addr = 10.6.14.1
[mon.ce-cl-102]
public_addr = 10.6.14.2
[mon.ce-cl-103]
public_addr = 10.6.14.3
Code:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class ssd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class ssd
device 7 osd.7 class ssd
device 8 osd.8 class ssd
device 9 osd.9 class ssd
device 10 osd.10 class ssd
device 11 osd.11 class ssd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host ce-cl-101 {
id -3 # do not change unnecessarily
id -4 class ssd # do not change unnecessarily
# weight 3.493
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.873
item osd.1 weight 0.873
item osd.2 weight 0.873
item osd.3 weight 0.873
}
host ce-cl-102 {
id -5 # do not change unnecessarily
id -6 class ssd # do not change unnecessarily
# weight 3.493
alg straw2
hash 0 # rjenkins1
item osd.4 weight 0.873
item osd.5 weight 0.873
item osd.6 weight 0.873
item osd.7 weight 0.873
}
host ce-cl-103 {
id -7 # do not change unnecessarily
id -8 class ssd # do not change unnecessarily
# weight 3.493
alg straw2
hash 0 # rjenkins1
item osd.8 weight 0.873
item osd.9 weight 0.873
item osd.10 weight 0.873
item osd.11 weight 0.873
}
root default {
id -1 # do not change unnecessarily
id -2 class ssd # do not change unnecessarily
# weight 10.479
alg straw2
hash 0 # rjenkins1
item ce-cl-101 weight 3.493
item ce-cl-102 weight 3.493
item ce-cl-103 weight 3.493
}
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
# end crush mapI am new to ceph and proxmox and this is for my final project, so please sorry any dumb statements i might written.
Thank you
Tobi
Last edited:
