Ceph BlockStorage Speed seems to limited

T0bi

New Member
Oct 14, 2021
6
0
1
20
Hi everyone,

I have a three node Proxmox VE Cluster to test Ceph for our enviroment.
Specs:
  • ASUS Server KNPA-U16 Series
  • 32 x AMD EPYC 7281 16-Core Processor (1 Socket)
  • 64GB RAM
  • two 1GB Nics (one used for to access the proxmox web GUI)
  • two 10GB Nics used for ceph-cluster (hosts connceted over a 10GB switch and tested the speed with iperf)
    • Network is seperated into Public Network and Cluster Network
    • Used in Bond mode access the Networks via VLAN (LACP is used)
  • 4x 1TB Samsung SSD (SAMSUNG_MZ7KM960HAHP-00005) directly plugged into the ASUS Chassis
After configuring the 12 OSDs I tested the Cluster with rados benchmark:
Code:
rados -p ceph_rbd2 bench 10 write                                                                  
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_ce-cl-101_613665
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       278       262   1047.92      1048   0.0634726   0.0588222
    2      16       575       559   1117.89      1188    0.040781   0.0565382
    3      16       847       831   1107.84      1088   0.0399402   0.0574625
    4      16      1126      1110   1109.74      1116   0.0372452   0.0569344
    5      16      1408      1392   1113.36      1128   0.0680987    0.057054
    6      16      1699      1683   1121.78      1164   0.0855339   0.0568148
    7      16      1989      1973   1127.22      1160   0.0954367   0.0565672
    8      16      2271      2255    1127.3      1128   0.0777163   0.0565807
    9      16      2572      2556    1135.8      1204   0.0575054   0.0562017
   10      16      2863      2847   1138.61      1164   0.0351853   0.0559141
Total time run:         10.0422
Total writes made:      2863
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     1140.39
Stddev Bandwidth:       47.1046
Max bandwidth (MB/sec): 1204
Min bandwidth (MB/sec): 1048
Average IOPS:           285
Stddev IOPS:            11.7762
Max IOPS:               301
Min IOPS:               262
Average Latency(s):     0.0559936
Stddev Latency(s):      0.0240188
Max latency(s):         0.288094
Min latency(s):         0.0220342
Cleaning up (deleting benchmark objects)
Removed 2863 objects

which seems to be ok for me
When I setup a vm on Proxmox using the RBD Storage, I tried to benchtest the read/write performance of the VM with fio:


Code:
fio --rw=write --name=test --size=5G --direct=1 --bs=1M
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=119MiB/s][w=119 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2317: Thu Oct 14 09:25:18 2021
  write: IOPS=117, BW=117MiB/s (123MB/s)(5120MiB/43645msec); 0 zone resets
    clat (usec): min=5998, max=19205, avg=8460.79, stdev=1176.34
     lat (usec): min=6072, max=19244, avg=8520.29, stdev=1176.09
    clat percentiles (usec):
     |  1.00th=[ 6390],  5.00th=[ 6718], 10.00th=[ 6980], 20.00th=[ 7439],
     | 30.00th=[ 7832], 40.00th=[ 8094], 50.00th=[ 8356], 60.00th=[ 8717],
     | 70.00th=[ 8979], 80.00th=[ 9372], 90.00th=[10028], 95.00th=[10552],
     | 99.00th=[11731], 99.50th=[12256], 99.90th=[13829], 99.95th=[14353],
     | 99.99th=[19268]
   bw (  KiB/s): min=112640, max=129024, per=99.97%, avg=120091.36, stdev=3620.92, samples=87
   iops        : min=  110, max=  126, avg=117.24, stdev= 3.56, samples=87
  lat (msec)   : 10=90.02%, 20=9.98%
  cpu          : usr=0.91%, sys=0.75%, ctx=5206, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5120,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=5120MiB (5369MB), run=43645-43645msec

Disk stats (read/write):
    dm-0: ios=0/5492, merge=0/0, ticks=0/46052, in_queue=46052, util=99.77%, aggrios=0/5348, aggrmerge=0/144, aggrticks=0/45036, aggrin_queue=34592, aggrutil=99.77%
  sda: ios=0/5348, merge=0/144, ticks=0/45036, in_queue=34592, util=99.77%

Code:
fio --rw=read --name=test --size=5G --direct=1 --bs=1M    
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=224MiB/s][r=224 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=2321: Thu Oct 14 09:26:26 2021
  read: IOPS=215, BW=216MiB/s (226MB/s)(5120MiB/23711msec)
    clat (msec): min=3, max=215, avg= 4.63, stdev= 3.00
     lat (msec): min=3, max=215, avg= 4.63, stdev= 3.00
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    6],
     | 99.00th=[    7], 99.50th=[    7], 99.90th=[    9], 99.95th=[   14],
     | 99.99th=[  215]
   bw (  KiB/s): min=131072, max=233472, per=100.00%, avg=221358.30, stdev=15084.94, samples=47
   iops        : min=  128, max=  228, avg=216.17, stdev=14.73, samples=47
  lat (msec)   : 4=14.94%, 10=84.96%, 20=0.06%, 50=0.02%, 250=0.02%
  cpu          : usr=0.24%, sys=1.64%, ctx=5155, majf=0, minf=268
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5120,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=216MiB/s (226MB/s), 216MiB/s-216MiB/s (226MB/s-226MB/s), io=5120MiB (5369MB), run=23711-23711msec

Disk stats (read/write):
    dm-0: ios=5110/4, merge=0/0, ticks=23192/12, in_queue=23204, util=99.60%, aggrios=5120/3, aggrmerge=0/1, aggrticks=23376/9, aggrin_queue=17064, aggrutil=99.56%
  sda: ios=5120/3, merge=0/1, ticks=23376/9, in_queue=17064, util=99.56%

The write performance seems very slow to me.

Also tested the blockstorage with fio from my promox server:

Code:
fio -ioengine=rbd -name=test -bs=1M -iodepth=1 -rw=randwrite -runtime=60 -pool=ceph_rbd -rbdname=test
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=rbd, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=122MiB/s][w=122 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=616821: Thu Oct 14 11:30:21 2021
  write: IOPS=117, BW=117MiB/s (123MB/s)(7046MiB/60005msec); 0 zone resets
    slat (usec): min=88, max=723, avg=154.15, stdev=34.38
    clat (msec): min=5, max=132, avg= 8.36, stdev= 1.78
     lat (msec): min=6, max=132, avg= 8.51, stdev= 1.78
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    7], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[    9],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   10], 95.00th=[   10],
     | 99.00th=[   11], 99.50th=[   12], 99.90th=[   13], 99.95th=[   14],
     | 99.99th=[  133]
   bw (  KiB/s): min=88064, max=126976, per=100.00%, avg=120298.49, stdev=4065.18, samples=119
   iops        : min=   86, max=  124, avg=117.48, stdev= 3.97, samples=119
  lat (msec)   : 10=95.98%, 20=4.00%, 250=0.01%
  cpu          : usr=2.05%, sys=0.14%, ctx=7064, majf=0, minf=1822
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7046,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=7046MiB (7388MB), run=60005-60005msec

Disk stats (read/write):
    dm-1: ios=0/1497, merge=0/0, ticks=0/568, in_queue=568, util=1.49%, aggrios=24/1040, aggrmerge=0/586, aggrticks=8/3350, aggrin_queue=3419, aggrutil=1.67%
  nvme0n1: ios=24/1040, merge=0/586, ticks=8/3350, in_queue=3419, util=1.67%

I also created a second rbd pool and ran the fio test on both pools simultaneously and both ended up with the same performance as shown before (not more that 127 Mb/s).
For me it seems that the speed is capped somehow.

Ceph Config:


Code:
global]
     mon_max_pg_per_osd = 400
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 10.8.16.1/16
     fsid = e9e5a603-7fc1-4ea7-9f84-04a632079588
     mon_allow_pool_delete = true
     mon_host = 10.6.14.1 10.6.14.2 10.6.14.3
     ms_bind_ipv4 = true
     ms_bind_ipv6 = false
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 10.6.14.1/16

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.ce-cl-101]
     host = ce-cl-101
     mds standby for name = pve

[mon.ce-cl-101]
     public_addr = 10.6.14.1

[mon.ce-cl-102]
     public_addr = 10.6.14.2

[mon.ce-cl-103]
     public_addr = 10.6.14.3

Code:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class ssd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class ssd
device 7 osd.7 class ssd
device 8 osd.8 class ssd
device 9 osd.9 class ssd
device 10 osd.10 class ssd
device 11 osd.11 class ssd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root

# buckets
host ce-cl-101 {
    id -3        # do not change unnecessarily
    id -4 class ssd        # do not change unnecessarily
    # weight 3.493
    alg straw2
    hash 0    # rjenkins1
    item osd.0 weight 0.873
    item osd.1 weight 0.873
    item osd.2 weight 0.873
    item osd.3 weight 0.873
}
host ce-cl-102 {
    id -5        # do not change unnecessarily
    id -6 class ssd        # do not change unnecessarily
    # weight 3.493
    alg straw2
    hash 0    # rjenkins1
    item osd.4 weight 0.873
    item osd.5 weight 0.873
    item osd.6 weight 0.873
    item osd.7 weight 0.873
}
host ce-cl-103 {
    id -7        # do not change unnecessarily
    id -8 class ssd        # do not change unnecessarily
    # weight 3.493
    alg straw2
    hash 0    # rjenkins1
    item osd.8 weight 0.873
    item osd.9 weight 0.873
    item osd.10 weight 0.873
    item osd.11 weight 0.873
}
root default {
    id -1        # do not change unnecessarily
    id -2 class ssd        # do not change unnecessarily
    # weight 10.479
    alg straw2
    hash 0    # rjenkins1
    item ce-cl-101 weight 3.493
    item ce-cl-102 weight 3.493
    item ce-cl-103 weight 3.493
}

# rules
rule replicated_rule {
    id 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit
}

# end crush map

I am new to ceph and proxmox and this is for my final project, so please sorry any dumb statements i might written.
Thank you :)

Tobi
 
Last edited:
Jan 16, 2018
223
47
33
The write bandwith is limited by network _and_ disk latency. A write can only be acknowledged after all replicas did acknowledge.


you say that it is your final project? Are you doing it as a project for a examination as Fachinformatiker?
 

T0bi

New Member
Oct 14, 2021
6
0
1
20
So is there any possibility to fix the latency. Because I am already using 10GB Network and SSD as recommended in the documentation.

Yes I do :)
 
Last edited:

T0bi

New Member
Oct 14, 2021
6
0
1
20
According to the https://proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09, I have the same speed as expected. Still confused how the slow write performance comes up
Code:
rados -p ceph_rbd bench 60 write -b 4M -t 16 --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_ce-cl-101_632723
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       297       281   1123.92      1124    0.025525   0.0556294
    2      16       585       569   1137.89      1152   0.0517181   0.0547435
    3      16       854       838   1117.19      1076   0.0400124   0.0568922
    4      16      1142      1126   1125.83      1152   0.0515378   0.0565423
    5      16      1432      1416   1132.62      1160   0.0604027   0.0560201
    6      16      1715      1699   1132.49      1132   0.0553688   0.0560493
    7      16      2008      1992    1138.1      1172   0.0715468   0.0559928
    8      16      2292      2276   1137.81      1136   0.0731875   0.0559049
    9      16      2576      2560   1137.58      1136   0.0657551   0.0560595
   10      16      2868      2852   1140.59      1168   0.0484576   0.0560073
   11      16      3162      3146    1143.8      1176   0.0419572   0.0556668
   12      16      3443      3427   1142.12      1124    0.056423   0.0558594
   13      16      3731      3715   1142.87      1152   0.0587582   0.0558914
   14      16      4022      4006   1144.37      1164   0.0427139   0.0558268
   15      16      4306      4290    1143.8      1136    0.039696   0.0557974
   16      16      4580      4564    1140.8      1096   0.0433688   0.0559013
   17      16      4874      4858   1142.86      1176   0.0340655   0.0559171
   18      16      5150      5134    1140.7      1104   0.0274726   0.0559111
   19      16      5447      5431   1143.17      1188   0.0401325   0.0558866
2021-10-15T07:33:58.403906+0200 min lat: 0.0199678 max lat: 0.330202 avg lat: 0.0558262
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      5739      5723    1144.4      1168   0.0380866   0.0558262
   21      16      6024      6008   1144.18      1140   0.0361727   0.0558386
   22      16      6303      6287    1142.9      1116   0.0326016   0.0558682
   23      16      6577      6561   1140.85      1096   0.0581289   0.0559835
   24      15      6859      6844   1140.47      1132   0.0270242   0.0560152
   25      16      7152      7136   1141.57      1168   0.0562496   0.0560259
   26      16      7427      7411   1139.95      1100   0.0766667   0.0560088
   27      16      7709      7693    1139.5      1128   0.0381044   0.0561083
   28      16      7984      7968   1138.09      1100   0.0313106   0.0560973
   29      16      8269      8253   1138.15      1140   0.0604682   0.0561477
   30      16      8565      8549   1139.66      1184   0.0297253   0.0560869
   31      16      8851      8835    1139.8      1144   0.0531363   0.0560911
   32      16      9137      9121   1139.92      1144   0.0779637   0.0560994
   33      16      9429      9413   1140.77      1168   0.0315354    0.056043
   34      16      9709      9693   1140.15      1120   0.0601145   0.0560696
   35      16     10003      9987   1141.17      1176    0.047494   0.0560259
   36      16     10298     10282   1142.25      1180   0.0831107   0.0559631
   37      16     10593     10577   1143.26      1180   0.0356994   0.0559245
   38      16     10887     10871   1144.12      1176   0.0542791   0.0558824
   39      16     11172     11156   1144.01      1140   0.0466315   0.0558784
2021-10-15T07:34:18.407331+0200 min lat: 0.0199678 max lat: 0.330202 avg lat: 0.0558507
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     11463     11447    1144.5      1164   0.0431687   0.0558507
   41      16     11764     11748   1145.95      1204   0.0385468   0.0558063
   42      16     12067     12051   1147.52      1212   0.0329523   0.0557283
   43      16     12363     12347   1148.36      1184   0.0584956   0.0556946
   44      16     12633     12617   1146.81      1080   0.0409819   0.0557537
   45      16     12921     12905   1146.92      1152   0.0568278   0.0557474
   46      16     13211     13195    1147.2      1160   0.0419724   0.0557398
   47      16     13485     13469   1146.11      1096   0.0289086   0.0557531
   48      16     13772     13756   1146.14      1148   0.0707775   0.0558029
   49      16     14068     14052   1146.91      1184   0.0608166   0.0557639
   50      16     14346     14330   1146.21      1112   0.0658696   0.0558024
   51      16     14637     14621   1146.55      1164   0.0356245    0.055783
   52      15     14924     14909   1146.65      1152   0.0554569   0.0557598
   53      16     15205     15189   1146.15      1120   0.0292565   0.0557814
   54      16     15498     15482   1146.63      1172   0.0698853   0.0557765
   55      16     15777     15761   1146.07      1116   0.0948062   0.0557893
   56      16     16065     16049   1146.17      1152   0.0268593   0.0557803
   57      16     16347     16331   1145.85      1128   0.0249169   0.0557859
   58      16     16634     16618   1145.88      1148   0.0281565   0.0558195
   59      16     16900     16884   1144.49      1064     0.05044   0.0558734
2021-10-15T07:34:38.410193+0200 min lat: 0.0194108 max lat: 0.448932 avg lat: 0.0559711
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      13     17157     17144   1142.75      1040   0.0719529   0.0559711
Total time run:         60.069
Total writes made:      17157
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     1142.49
Stddev Bandwidth:       34.8079
Max bandwidth (MB/sec): 1212
Min bandwidth (MB/sec): 1040
Average IOPS:           285
Stddev IOPS:            8.70197
Max IOPS:               303
Min IOPS:               260
Average Latency(s):     0.0559838
Stddev Latency(s):      0.0278926
Max latency(s):         0.448932
Min latency(s):         0.0194108
Code:
rados -p ceph_rbd bench 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       349       333    1331.7      1332   0.0233486    0.044806
    2      16       689       673   1345.46      1360   0.0160862   0.0459491
    3      16      1015       999   1331.59      1304   0.0189319   0.0458234
    4      15      1316      1301   1300.29      1208   0.0163352    0.047809
    5      15      1651      1636   1308.06      1340    0.142338   0.0473017
    6      16      1948      1932   1287.37      1184   0.0163332   0.0482619
    7      16      2303      2287   1306.29      1420     0.02681   0.0477911
    8      16      2662      2646   1322.47      1436   0.0386918   0.0473118
    9      15      3042      3027   1344.74      1524   0.0196303   0.0464932
   10      15      3419      3404   1361.01      1508    0.106765   0.0459653
   11      16      3804      3788    1376.9      1536   0.0173765   0.0454185
   12      16      4130      4114   1370.81      1304   0.0604451   0.0455623
   13      15      4431      4416   1358.21      1208   0.0216385   0.0460546
   14      15      4761      4746   1355.39      1320   0.0180358   0.0460475
   15      16      5058      5042   1343.96      1184   0.0142623   0.0464101
   16      16      5381      5365    1340.7      1292   0.0327717   0.0467381
   17      16      5720      5704   1341.58      1356   0.0187207    0.046547
   18      16      6035      6019   1337.04      1260     0.22298   0.0468064
   19      15      6324      6309   1327.69      1160   0.0232584   0.0471489
2021-10-15T07:35:45.763740+0200 min lat: 0.0134656 max lat: 0.39685 avg lat: 0.0472598
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      6619      6603    1320.1      1176   0.0177279   0.0472598
   21      16      6931      6915   1316.66      1248    0.139139   0.0475118
   22      16      7292      7276   1322.44      1444   0.0981865    0.047384
   23      15      7626      7611   1323.15      1340   0.0163898   0.0473809
   24      15      8000      7985   1330.34      1496   0.0191901    0.047103
   25      16      8361      8345   1334.72      1440   0.0284332   0.0469593
   26      16      8696      8680   1334.92      1340   0.0151112    0.046964
   27      16      9042      9026   1336.72      1384    0.029419   0.0468722
   28      15      9429      9414    1344.4      1552   0.0631867   0.0466559
   29      16      9786      9770   1347.14      1424   0.0165838    0.046536
   30      16     10121     10105   1346.88      1340   0.0216874   0.0465052
   31      16     10457     10441   1346.78      1344   0.0761354   0.0465736
   32      16     10777     10761   1344.69      1280   0.0274222   0.0466374
   33      15     11094     11079   1342.47      1272    0.128138    0.046718
   34      16     11440     11424   1343.57      1380   0.0485888   0.0466963
   35      16     11775     11759   1343.46      1340   0.0200051   0.0466428
   36      15     12094     12079   1341.68      1280   0.0314515   0.0467564
   37      16     12451     12435    1343.9      1424   0.0480313   0.0466893
   38      16     12793     12777   1344.53      1368   0.0179664   0.0466743
   39      15     13135     13120    1345.2      1372    0.016623   0.0466221
2021-10-15T07:36:05.769037+0200 min lat: 0.0134656 max lat: 0.39685 avg lat: 0.0466418
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     13478     13462   1345.77      1368    0.055008   0.0466418
   41      16     13797     13781   1344.06      1276   0.0174964   0.0466914
   42      15     14127     14112   1343.57      1324   0.0198883   0.0467134
   43      16     14484     14468   1345.44      1424   0.0152019   0.0466732
   44      16     14847     14831   1347.85      1452   0.0165147   0.0465779
   45      15     15217     15202   1350.86      1484   0.0342093     0.04646
   46      16     15559     15543   1351.14      1364   0.0425679   0.0464816
   47      16     15920     15904   1353.11      1444    0.136798   0.0463956
   48      16     16229     16213   1350.67      1236   0.0203823   0.0464661
   49      15     16603     16588   1353.71      1500   0.0168809   0.0463788
   50      16     16943     16927   1353.75      1356   0.0152585   0.0463716
Total time run:       50.6316
Total reads made:     17157
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1355.44
Average IOPS:         338
Stddev IOPS:          24.8647
Max IOPS:             388
Min IOPS:             290
Average Latency(s):   0.0463301
Max latency(s):       0.39685
Min latency(s):       0.0134656
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,502
1,123
164
try single-thread performance (your benchmark is using 16 threads currently).
 

jsterr

Active Member
Jul 24, 2020
153
27
28
30
Can u name the "bad performance" are u referring to the rados bench? to your fio from the initial posts? Pls name the benchmark which shows "bad" performance . Please also post VM-Config if your referring to the fio benchmark. EDIT: ah your doing fio directly to the rbd pool.

If your referring to this, play around with higher iodepth and use higher blocksize if you wanna get good values (might not be real workload though)

Code:
fio -ioengine=rbd -name=test -bs=1M -iodepth=1 -rw=randwrite -runtime=60 -pool=ceph_rbd -rbdname=test
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
6,502
1,123
164

T0bi

New Member
Oct 14, 2021
6
0
1
20
Thanks for your replys, I understand the problem I have made. Also https://www.thomas-krenn.com/de/wiki/Fio_Grundlagen helped me to understand better about working with fio

After increasing the iodepth to 16 I get these results:
Code:
fio -ioengine=rbd -name=test -bs=1M -iodepth=16 -rw=randwrite -runtime=60 -pool=ceph_rbd -rbdname=test
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=rbd, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=853MiB/s][w=852 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=709073: Fri Oct 15 09:49:59 2021
  write: IOPS=913, BW=914MiB/s (958MB/s)(8142MiB/8911msec); 0 zone resets
    slat (usec): min=160, max=2365, avg=409.68, stdev=159.46
    clat (msec): min=7, max=236, avg=17.08, stdev=12.30
     lat (msec): min=7, max=237, avg=17.49, stdev=12.30
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   20], 90.00th=[   24], 95.00th=[   29],
     | 99.00th=[   70], 99.50th=[   84], 99.90th=[  218], 99.95th=[  226],
     | 99.99th=[  236]
   bw (  KiB/s): min=856064, max=1001472, per=100.00%, avg=944609.88, stdev=40751.66, samples=17
   iops        : min=  836, max=  978, avg=922.47, stdev=39.80, samples=17
  lat (msec)   : 10=6.01%, 20=75.98%, 50=16.58%, 100=1.11%, 250=0.33%
  cpu          : usr=37.59%, sys=1.39%, ctx=4816, majf=0, minf=25351
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,8142,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=914MiB/s (958MB/s), 914MiB/s-914MiB/s (958MB/s-958MB/s), io=8142MiB (8538MB), run=8911-8911msec

Disk stats (read/write):
    dm-1: ios=0/206, merge=0/0, ticks=0/20, in_queue=20, util=1.47%, aggrios=4/132, aggrmerge=0/75, aggrticks=2/27, aggrin_queue=37, aggrutil=1.64%
  nvme0n1: ios=4/132, merge=0/75, ticks=2/27, in_queue=37, util=1.64%

Thank you all for your help :)

For my final project I wanted to test our storage (my new ceph cluster and our SAN).
Therefore I created 2 VMs. One is hosted on ceph and one hosted on the SAN.
Which fio test would you recommend to get an meaningful result for the documentation ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!