ceph performance 4node all NVMe 56GBit Ethernet

G

Gerhard W. Recher

Well-Known Member
Mar 10, 2017
158
8
58
Munich
Hi Folks,

Whis performanche shoud i expect for this cluster ? are my settings ok ?

4 nodes:
Code: 
system: Supermicro 2028U-TN24R4T+

2 port Mellanox connect x3pro 56Gbit
4 port intel 10GigE
memory: 768 GBytes
CPU DUAL  Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

ceph: 28 osds
24  Intel Nvme 2000GB Intel SSD DC P3520, 2,5", PCIe 3.0 x4,
 4  Intel Nvme 1,6TB Intel SSD DC P3700, 2,5", U.2 PCIe 3.0

Version:
Code: 
 pveversion  -v
proxmox-ve: 5.0-12 (running kernel: 4.10.15-1-pve)
pve-manager: 5.0-10 (running version: 5.0-10/0d270679)
pve-kernel-4.10.15-1-pve: 4.10.15-12
pve-kernel-4.10.11-1-pve: 4.10.11-9
libpve-http-server-perl: 2.0-4
lvm2: 2.02.168-pve2
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-9
qemu-server: 5.0-4
pve-firmware: 2.0-2
libpve-common-perl: 5.0-12
libpve-guest-common-perl: 2.0-1
libpve-access-control: 5.0-4
libpve-storage-perl: 5.0-3
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-2
pve-qemu-kvm: 2.9.0-2
pve-container: 2.0-6
pve-firewall: 3.0-1
pve-ha-manager: 2.0-1
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-2
lxcfs: 2.0.7-pve2
criu: 2.11.1-1~bpo90
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.9-pve16~bpo90
ceph: 12.0.3-pve2

corosync: ring0 56Gbit ring1 10Gbit
Code: 
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pve02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.221.142
    ring1_addr: 192.168.111.142
  }

  node {
    name: pve04
    nodeid: 4
    quorum_votes: 1
    ring0_addr: 192.168.221.144
    ring1_addr: 192.168.111.144
  }

  node {
    name: pve01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.221.141
     ring1_addr: 192.168.111.141
 }

  node {
    name: pve03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.221.143
    ring1_addr: 192.168.111.143
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: net4sec01
  config_version: 5
  ip_version: ipv4
  netmtu: 9000
  rrp_mode: passive
  secauth: on
  version: 2
  interface {
    bindnetaddr: 192.168.221.0
    ringnumber: 0
  }
  interface {
    bindnetaddr: 192.168.111.0
    ringnumber: 1
  }

}

ceph.conf
Code: 
[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         cluster network = 192.168.100.0/24
         filestore xattr use omap = true
         fsid = cb0aba69-bad9-4d30-b163-c19f0fd1ec53
         keyring = /etc/pve/priv/$cluster.$name.keyring
         osd journal size = 57344

         osd pool default min size = 2
         public network = 192.168.100.0/24


rbdreadaheaddisable after bytes = 0
rbdreadaheadmax bytes = 4194304
mon compact on trim = false
osdpgbits = 8
osdpgpbits = 8
mon pgwarn max object skew = 100000
mon pgwarn min per osd= 0
mon pgwarn max per osd= 32768
mon_allow_pool_delete=true
mon allow pool delete = true
debug_lockdep= 0/0
debug_context= 0/0
debug_crush= 0/0
debug_buffer= 0/0
debug_timer= 0/0
debug_filer= 0/0
debug_objecter= 0/0
debug_rados= 0/0
debug_rbd= 0/0
debug_ms= 0/0
debug_monc= 0/0
debug_tp= 0/0
debug_auth= 0/0
debug_finisher= 0/0
debug_heartbeatmap= 0/0
debug_perfcounter= 0/0
debug_asok= 0/0
debug_throttle= 0/0
debug_mon= 0/0
debug_paxos= 0/0
debug_rgw= 0/0
perf = true
mutex_perf_counter= true
throttler_perf_counter= false
rbdcache = false

[mon]
mon_allow_pool_delete=true

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring
#osd_mount_options_xfs= rw,noatime,inode64,logbsize=256k,delaylog
#osd_mkfs_options_xfs= -f -isize=2048
osd_op_threads= 32
filestore_queue_max_ops=5000
filestore_queue_committing_max_ops=5000
journal_max_write_entries=1000
journal_queue_max_ops=3000
objecter_inflight_ops=102400
filestore_wbthrottle_enable=false
filestore_queue_max_bytes=1048576000
filestore_queue_committing_max_bytes=1048576000
journal_max_write_bytes=1048576000
journal_queue_max_bytes=1048576000
ms_dispatch_throttle_bytes=1048576000
objecter_infilght_op_bytes=1048576000
osd_mkfs_type= xfs
filestore_max_sync_interval=10
osd_client_message_size_cap= 0
osd_client_message_cap= 0
osd_enable_op_tracker= false
filestore_fd_cache_size= 128
filestore_fd_cache_shards= 32
filestore_op_threads= 6

[mon.2]
         host = pve03
         mon addr = 192.168.100.143:6789

[mon.3]
         host = pve04
         mon addr = 192.168.100.144:6789

[mon.0]
         host = pve01
         mon addr = 192.168.100.141:6789

[mon.1]
         host = pve02
         mon addr = 192.168.100.142:6789


storage.conf
Code: 
dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

rbd: vmstorage
        monhost 192.168.100.141 192.168.100.142 192.168.100.143 192.168.100.144
        content images
        krbd 0
        pool vmpool
        username admin

rbd: lxcstorage
        monhost 192.168.100.141 192.168.100.142 192.168.100.143 192.168.100.144
        content rootdir
        krbd 1
        pool lxcpool
        username admin
Code: 
 rados bench -p test 60 write --no-cleanup -t 256
hints = 1
Maintaining 256 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_21330
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1     255       587       332   1327.89      1328    0.401936    0.451267
    2     255      1232       977   1953.77      2580    0.431364    0.416914
    3     255      1808      1553   2070.42      2304    0.521574    0.414513
    4     255      2290      2035   2034.76      1928     0.42955    0.449002
    5     255      2945      2690   2151.74      2620    0.409057     0.43506
    6     255      3584      3329   2219.07      2556    0.371282    0.431392
    7     255      4282      4027   2300.87      2792    0.356222    0.419709
    8     255      4938      4683   2341.21      2624      0.3885    0.414551
    9     255      5586      5331   2369.04      2592    0.366132    0.413349
   10     255      6258      6003   2400.91      2688    0.404704    0.408512
   11     255      6871      6616   2405.52      2452    0.440065    0.409025
   12     255      7548      7293    2430.7      2708    0.364629    0.407135
   13     255      8202      7947   2444.93      2616      0.3959    0.405625
   14     255      8880      8625   2463.98      2712    0.338021    0.403908
   15     255      9544      9289   2476.76      2656    0.384629    0.402066
   16     255     10207      9952    2487.7      2652    0.392296    0.401327
   17     255     10874     10619   2498.29      2668    0.365575    0.400458
   18     255     11562     11307   2512.37      2752    0.397945    0.398371
   19     255     12220     11965   2518.65      2632    0.406351    0.397918
2017-06-28 17:48:53.009115 min lat: 0.328529 max lat: 1.13894 avg lat: 0.397086
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20     255     12900     12645    2528.7      2720    0.342094    0.397086
   21     255     13549     13294   2531.87      2596    0.468214    0.396117
   22     255     14146     13891   2525.32      2388    0.435823    0.397572
   23     255     14816     14561   2532.03      2680    0.374709    0.397402
   24     255     15456     15201   2533.18      2560    0.433976    0.397087
   25     255     16098     15843   2534.56      2568    0.397797    0.397367
   26     255     16743     16488    2536.3      2580    0.416258    0.397288
   27     255     17415     17160   2541.91      2688    0.376516    0.396634
   28     255     18057     17802   2542.83      2568    0.394027    0.396856
   29     255     18760     18505    2552.1      2812    0.343443    0.395895
   30     255     19445     19190   2558.35      2740    0.403017    0.394682
   31     255     20142     19887   2565.75      2788    0.363161     0.39397
   32     255     20767     20512   2563.68      2500    0.419039    0.393867
   33     255     21416     21161   2564.65      2596     0.46306    0.393661
   34     255     22061     21806   2565.09      2580     0.38955    0.394255
   35     255     22729     22474   2568.14      2672    0.413944    0.393893
   36     255     23423     23168    2573.9      2776    0.338478    0.393497
   37     255     24066     23811   2573.84      2572    0.385596    0.393484
   38     255     24724     24469   2575.37      2632    0.413361    0.393143
   39     255     25372     25117   2575.78      2592    0.395716    0.393376
2017-06-28 17:49:13.011688 min lat: 0.304412 max lat: 1.13894 avg lat: 0.392676
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40     255     26078     25823   2581.98      2824    0.353164    0.392676
   41     255     26704     26449   2580.07      2504    0.424294    0.392792
   42     255     27356     27101   2580.73      2608    0.400599    0.392743
   43     255     28082     27827   2588.23      2904    0.355009    0.392025
   44     255     28662     28407   2582.13      2320    0.401303    0.392872
   45     255     29354     29099   2586.25      2768    0.382967    0.392415
   46     255     30019     29764   2587.85      2660    0.358233     0.39227
   47     255     30730     30475   2593.29      2844    0.337088    0.391703
   48     255     31418     31163   2596.59      2752    0.381043    0.391181
   49     255     32126     31871   2601.39      2832    0.334852    0.390555
   50     255     32820     32565   2604.88      2776    0.369552    0.389977
   51     255     33512     33257   2608.07      2768    0.385844    0.389564
   52     255     34191     33936   2610.14      2716    0.399073    0.389089
   53     255     34863     34608    2611.6      2688    0.369976    0.389156
   54     255     35555     35300   2614.49      2768    0.375974    0.388742
   55     255     36300     36045   2621.13      2980    0.341197    0.387948
   56     255     36951     36696   2620.82      2604    0.376959    0.387906
   57     255     37608     37353   2620.94      2628    0.400293    0.387795
   58     255     38364     38109   2627.88      3024     0.31129    0.387138
   59     255     39079     38824   2631.81      2860    0.362062    0.386474
2017-06-28 17:49:33.014153 min lat: 0.0391196 max lat: 1.13894 avg lat: 0.386314
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60     254     39760     39506   2633.41      2728   0.0391196    0.386314
Total time run:         60.221536
Total writes made:      39760
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     2640.92
Stddev Bandwidth:       238.636
Max bandwidth (MB/sec): 3024
Min bandwidth (MB/sec): 1328
Average IOPS:           660
Stddev IOPS:            59
Max IOPS:               756
Min IOPS:               332
Average Latency(s):     0.386232
Stddev Latency(s):      0.0404413
Max latency(s):         1.13894
Min latency(s):         0.0391196


root@pve01:~# rados bench -p test 60 seq --no-cleanup -t 256
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1     255       632       377   1507.56      1508    0.494802    0.427645
    2     255      1230       975   1948.87      2392    0.414537    0.422791
    3     255      1732      1477   1968.15      2008    0.591348    0.450344
    4     255      2118      1863    1862.1      1544    0.637502    0.491053
    5     255      2841      2586   2067.79      2892     0.37308    0.461835
    6     255      3487      3232   2153.24      2584    0.387218    0.449526
    7     255      3996      3741   2136.47      2036    0.434377    0.451607
    8     255      4752      4497   2247.32      3024    0.324395    0.437766
    9     255      5434      5179   2300.25      2728    0.419009    0.425017
   10     255      6139      5884   2352.09      2820    0.252401    0.422692
   11     255      6776      6521   2369.62      2548    0.378103     0.41804
   12     255      7538      7283   2425.92      3048    0.256583    0.411665
   13     255      8260      8005   2461.41      2888    0.324949    0.406188
   14     255      8888      8633   2464.99      2512    0.491539    0.403485
   15     255      9392      9137   2435.05      2016    0.518047    0.403731
   16     255     10030      9775   2442.14      2552    0.349451    0.410652
   17     255     10468     10213   2401.51      1752    0.652172    0.412435
   18     255     11364     11109   2467.15      3584    0.263525    0.407514
   19     255     12085     11830   2488.89      2884    0.324932    0.404056
2017-06-28 17:54:54.618910 min lat: 0.240869 max lat: 1.07813 avg lat: 0.400013
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20     255     12800     12545   2507.41      2860    0.478608    0.400013
   21     255     13537     13282    2528.3      2948    0.310006     0.39846
   22     255     14185     13930   2531.14      2592    0.488355    0.396167
   23     255     15083     14828   2577.22      3592     0.25903    0.391744
   24     255     15879     15624   2602.39      3184    0.338573    0.386972
   25     255     16610     16355   2615.23      2924    0.293421    0.386109
   26     255     17303     17048   2621.25      2772    0.356902    0.385007
   27     255     18028     17773   2631.49      2900    0.312596    0.384045
   28     255     18814     18559   2649.77      3144    0.321358    0.381201
   29     255     19468     19213    2648.6      2616    0.367013     0.38129
   30     255     20084     19829   2642.33      2464    0.304318    0.382964
   31     255     20823     20568    2652.4      2956    0.307515    0.381572
   32     255     21491     21236    2652.6      2672    0.418097    0.380527
   33     255     22094     21839   2645.14      2412    0.536086    0.381787
   34     255     22857     22602   2657.06      3052    0.292721    0.381267
   35     255     23544     23289   2659.63      2748    0.339959    0.380789
   36     255     24093     23838   2646.73      2196      0.5003    0.382153
   37     255     24660     24405   2636.46      2268    0.504696    0.383727
   38     255     25327     25072   2637.28      2668    0.310324    0.384147
   39     255     25940     25685   2632.49      2452    0.303782     0.38523
2017-06-28 17:55:14.634055 min lat: 0.240869 max lat: 1.07813 avg lat: 0.383285
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40     255     26724     26469   2645.06      3136    0.338664    0.383285
   41     255     27489     27234   2655.13      3060    0.363139    0.381793
   42     255     28074     27819   2647.62      2340    0.379982    0.382905
   43     255     28804     28549   2653.86      2920    0.299086    0.382458
   44     255     29442     29187   2651.54      2552    0.373616    0.382691
   45     255     30244     29989   2663.89      3208    0.286097      0.3812
   46     255     30992     30737   2671.01      2992    0.304721    0.380291
   47     255     31535     31280   2660.38      2172    0.509976    0.381067
   48     255     32129     31874   2654.31      2376    0.465523    0.382397
   49     255     32885     32630   2661.84      3024    0.359988    0.381404
   50     255     33598     33343   2665.64      2852     0.31678    0.381185
   51     255     34233     33978   2663.17      2540    0.449024      0.3814
   52     255     35108     34853   2679.22      3500    0.311267    0.379436
   53     255     35816     35561   2682.07      2832    0.336318    0.378696
   54     255     36474     36219   2681.14      2632    0.420253    0.378818
   55     255     37092     36837   2677.33      2472    0.424767    0.379128
   56     255     37922     37667   2688.73      3320    0.256273    0.378256
   57     255     38657     38402   2693.09      2940    0.314589    0.377511
   58     255     39383     39128    2696.7      2904    0.405418    0.376766
   59      82     39760     39678   2688.28      2200    0.562121    0.377946
Total time run:       59.099695
Total reads made:     39760
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   2691.05
Average IOPS:         672
Stddev IOPS:          111
Max IOPS:             898
Min IOPS:             377
Average Latency(s):   0.378223
Max latency(s):       1.15026
Min latency(s):       0.0387909
root@pve01:~#
 
performance benchmarks looks not bad. would be interesting to see your results if you run your OSDs with bluestore.
(beginning next week, we publish 12.1 rc1 packages).
 
Tom

Wich steps must be done?

The plan to go into production in very short time frame

Gerhard
 
Hi,
If you disable the ceph auth you should use the firewall and block this ports on all interfaces and only allow the members ip.
disable auth will bring definitely more speed.

If you have 2 10GB nics free I would consider to separate public and private ceph network.
 
hmmm... set auth to none .... but know all vm's don't start anymore ! how to change startupcmds ?
 
You have to restart all services like osd, mon and mgr if enabled
 
i restarted the whole cluster ....

Code: 
/usr/bin/kvm -id 102 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=45631093-fe88-4493-b759-297dd584725a' -name win2003 -smp '8,sockets=2,cores=4,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga cirrus -vnc unix:/var/run/qemu-server/102.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 8096 -k de -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -chardev 'socket,path=/var/run/qemu-server/102.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ec3befda203a' -drive 'file=/var/lib/vz/template/iso/virtio-win-0.1.126.iso,if=none,id=drive-ide0,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200' -drive 'file=rbd:vmpool/vm-102-disk-2:mon_host=192.168.100.141;192.168.100.142;192.168.100.143;192.168.100.144:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/vmstorage.keyring,if=none,id=drive-ide1,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1' -drive 'file=/var/lib/vz/template/iso/win_aio_de.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=201' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=52:62:47:C7:DB:2B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime'
 
you have also edit your storage.cfg

remove the user line
 
i removed the username line, but gui still shows user:admin ...
 
can you please post the storage.cfg
 
root@pve01:/etc/pve# cat storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

rbd: vmstorage
monhost 192.168.100.141 192.168.100.142 192.168.100.143 192.168.100.144
content images
krbd 0
pool vmpool

rbd: lxcstorage
monhost 192.168.100.141 192.168.100.142 192.168.100.143 192.168.100.144
content rootdir
krbd 1
pool lxcpool
 
gui view of this ...
 

Attachments

  • storage.PNG
    storage.PNG
    35.6 KB · Views: 48
I see this is only in the GUI it is already fixed update the system and is away
 
no updates avail ... if i start kvm manually

... id=admin:auth_supported=none ....
vm starts, but left tree has no update and summary state: error despite this instance is up and running :(
 

Attachments

  • vmstart.PNG
    vmstart.PNG
    153 KB · Views: 32
Sorry my fault.
We do not push it yet to the repo.
 
i will stay tuned ...

how to restart all 4 nodes with one command ? to keep ceph happy ?
 
The wiki will be updated when luminous is released.
 
tom said:
performance benchmarks looks not bad. would be interesting to see your results if you run your OSDs with bluestore.
(beginning next week, we publish 12.1 rc1 packages).
Click to expand...
same cluster but now bluestore, Tom!
some 10% better performance!
I think with RDMA enabled we will gain another 10% to 20% speedup
but I still have no glue how to accomplish RDAM with ceph and underlying 56 Gbit Mellanox 3x cards ....


Code: 
rados bench -p test 60 write --no-cleanup -t 256
hints = 1
Maintaining 256 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve04_7263
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1     255       695       440   1759.86      1760    0.322953    0.375324
    2     255      1461      1206   2411.68      3064    0.285039    0.354101
    3     255      2091      1836   2447.68      2520    0.290834    0.364262
    4     255      2905      2650   2649.67      3256    0.369677    0.349326
    5     255      3690      3435   2747.63      3140    0.325579    0.348477
    6     255      4510      4255   2836.29      3280    0.328874    0.339814
    7     255      5321      5066   2894.48      3244    0.319146    0.336551
    8     255      6141      5886   2942.62      3280     0.31255    0.332905
    9     255      6884      6629   2945.85      2972    0.320519    0.334749
   10     255      7654      7399   2959.23      3080    0.379352    0.333287
   11     255      8484      8229      2992      3320      0.2956    0.331771
   12     255      9238      8983   2993.98      3016    0.330701    0.331918
   13     255     10066      9811   3018.42      3312    0.323461    0.330297
   14     255     10782     10527   3007.37      2864    0.398946    0.330939
   15     255     11571     11316   3017.25      3156    0.308119    0.331737
   16     255     12289     12034   3008.15      2872    0.359057    0.332061
   17     255     13105     12850   3023.18      3264    0.275715    0.332167
   18     255     13818     13563   3013.65      2852    0.326248    0.333292
   19     255     14615     14360   3022.81      3188    0.306512     0.33273
2017-09-11 15:52:38.320174 min lat: 0.266921 max lat: 0.779452 avg lat: 0.331503
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20     255     15439     15184   3036.46      3296    0.333295    0.331503
   21     255     16316     16061   3058.86      3508    0.306362    0.329427
   22     255     17071     16816   3057.07      3020    0.374988    0.329457
   23     255     17843     17588    3058.4      3088    0.308725     0.32981
   24     255     18600     18345   3057.12      3028    0.291003     0.33029
   25     255     19405     19150   3063.61      3220    0.309644    0.329756
   26     255     20159     19904   3061.77      3016    0.353388    0.329617
   27     255     20984     20729   3070.58      3300    0.285259    0.329363
   28     255     21747     21492    3069.9      3052    0.396364    0.329098
   29     255     22540     22285   3073.41      3172    0.300014     0.32927
   30     255     23339     23084   3077.49      3196    0.314734    0.328986
   31     255     24148     23893   3082.59      3236     0.28478    0.328673
   32     255     24889     24634   3078.87      2964    0.369973    0.328499
   33     255     25690     25435   3082.65      3204     0.36503     0.32849
   34     255     26401     26146   3075.63      2844     0.37617    0.329192
   35     255     27169     26914   3075.51      3072    0.342169    0.329485
   36     255     27953     27698   3077.18      3136    0.311246    0.329636
   37     255     28695     28440   3074.22      2968     0.28766    0.330079
   38     255     29494     29239   3077.42      3196    0.324348    0.329848
   39     255     30243     29988   3075.32      2996    0.318523    0.329978
2017-09-11 15:52:58.322677 min lat: 0.266921 max lat: 0.79825 avg lat: 0.331278
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40     255     30882     30627   3062.34      2556    0.363377    0.331278
   41     255     31647     31392   3062.27      3060    0.307919     0.33167
   42     255     32344     32089   3055.73      2788    0.316628    0.332418
   43     255     33060     32805   3051.27      2864    0.393986    0.332769
   44     255     33828     33573   3051.73      3072    0.388327      0.3326
   45     255     34524     34269   3045.78      2784    0.337606    0.333454
   46     255     35258     35003   3043.38      2936    0.350715    0.333809
   47     255     36020     35765   3043.47      3048    0.294253    0.334064
   48     255     36734     36479   3039.56      2856    0.365727    0.334415
   49     255     37520     37265   3041.69      3144    0.298278     0.33435
   50     255     38382     38127    3049.8      3448    0.296466    0.333525
   51     255     39011     38756   3039.33      2516    0.401989    0.334326
   52     255     39806     39551   3042.03      3180    0.337086     0.33439
   53     255     40560     40305   3041.53      3016    0.343938    0.334439
   54     255     41312     41057   3040.91      3008    0.376941    0.334528
   55     255     42151     41896   3046.63      3356    0.289804    0.334104
   56     255     42972     42717   3050.86      3284    0.341379    0.333443
   57     255     43727     43472   3050.31      3020    0.358882    0.333586
   58     255     44541     44286   3053.85      3256    0.359234    0.333164
   59     255     45329     45074   3055.51      3152    0.297052    0.333237
2017-09-11 15:53:18.324795 min lat: 0.0440912 max lat: 0.820582 avg lat: 0.333008
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60     246     46112     45866   3057.38      3168   0.0440912    0.333008
Total time run:         60.258160
Total writes made:      46112
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     3060.96
Stddev Bandwidth:       265.955
Max bandwidth (MB/sec): 3508
Min bandwidth (MB/sec): 1760
Average IOPS:           765
Stddev IOPS:            66
Max IOPS:               877
Min IOPS:               440
Average Latency(s):     0.333311
Stddev Latency(s):      0.0414064
Max latency(s):         0.820582
Min latency(s):         0.0440912
root@pve04:/home# rados bench -p test 60 seq --no-cleanup -t 256
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1     255       830       575    2295.6      2300    0.282731    0.301628
    2     255      1437      1182   2360.73      2428    0.569517    0.324787
    3     255      2277      2022   2693.42      3360    0.319415    0.338516
    4     255      2825      2570   2567.36      2192    0.550117    0.361093
    5     255      3544      3289   2628.07      2876    0.425397    0.360862
    6     255      4267      4012    2671.7      2892    0.312231    0.361865
    7     255      5100      4845   2765.89      3332    0.233643    0.353555
    8     255      5803      5548   2771.54      2812    0.305028    0.353571
    9     255      6640      6385   2835.47      3348    0.342712    0.347334
   10     255      7300      7045   2815.91      2640    0.261581    0.352346
   11     255      8025      7770   2823.02      2900    0.435036    0.348934
   12     255      8721      8466   2819.67      2784    0.280462    0.352853
   13     255      9588      9333   2869.35      3468    0.346929    0.346958
   14     255     10236      9981   2849.32      2592    0.442593    0.347858
   15     255     10886     10631   2832.69      2600     0.39731    0.351129
   16     255     11630     11375   2841.28      2976    0.488922    0.350188
   17     255     12375     12120   2849.36      2980     0.29826    0.351573
   18     255     13173     12918   2868.32      3192    0.416224    0.349602
   19     255     14102     13847   2912.48      3716    0.262994    0.345065
2017-09-11 15:54:18.215947 min lat: 0.212607 max lat: 0.87161 avg lat: 0.345059
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20     255     14834     14579   2913.17      2928    0.297634    0.345059
   21     255     15807     15552   2959.42      3892    0.303882    0.340267
   22     255     16379     16124   2928.79      2288    0.505752    0.341849
   23     255     17211     16956   2946.12      3328    0.343587    0.342301
   24     255     17990     17735   2953.15      3116    0.272775    0.341718
   25     255     18566     18311   2927.19      2304    0.439569    0.343065
   26     255     19380     19125   2939.81      3256    0.308609    0.343398
   27     255     20158     19903   2945.73      3112    0.311033    0.342667
   28     255     20887     20632   2944.65      2916    0.381386    0.342425
   29     255     21705     21450   2955.91      3272    0.221929    0.342339
   30     255     22598     22343   2976.42      3572    0.300477    0.339718
   31     255     23382     23127   2981.55      3136     0.29159    0.339163
   32     255     24046     23791   2971.38      2656    0.357949    0.339853
   33     255     24720     24465   2963.03      2696    0.373559    0.341134
   34     255     25387     25132   2954.35      2668    0.423982    0.341826
   35     255     26160     25905   2958.27      3092    0.367056     0.34194
   36     255     26756     26501   2942.29      2384    0.489899    0.343336
   37     255     27318     27063   2923.49      2248    0.368831    0.346348
   38     255     28142     27887   2933.26      3296     0.25891    0.345682
   39     255     28797     28542   2925.22      2620    0.419238     0.34569
2017-09-11 15:54:38.229079 min lat: 0.212607 max lat: 1.01318 avg lat: 0.346141
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40     255     29573     29318   2929.51      3104    0.361157    0.346141
   41     255     30339     30084   2932.78      3064    0.326779    0.345587
   42     255     31104     30849    2935.7      3060    0.431067    0.345072
   43     255     31939     31684   2944.93      3340    0.222381    0.344732
   44     255     32916     32661   2966.79      3908    0.239926    0.342255
   45     255     33726     33471   2972.84      3240    0.235326    0.341507
   46     255     34465     34210   2972.47      2956    0.386583    0.340958
   47     255     35453     35198   2993.25      3952     0.24847    0.339269
   48     255     36161     35906   2989.71      2832    0.281045    0.339474
   49     255     36701     36446   2972.78      2160    0.517341    0.340498
   50     255     37530     37275   2979.62      3316     0.25064    0.340869
   51     255     38376     38121   2987.54      3384    0.287837    0.339988
   52     255     39112     38857   2986.51      2944    0.262252    0.340163
   53     255     39888     39633   2988.72      3104    0.375201    0.339327
   54     255     40721     40466   2995.02      3332    0.300606    0.339084
   55     255     41423     41168   2991.61      2808    0.318426    0.339549
   56     255     42063     41808   2983.88      2560    0.363967    0.340366
   57     255     42836     42581   2985.75      3092     0.27237    0.340405
   58     255     43561     43306   2984.26      2900    0.384015    0.339966
   59     255     44331     44076   2985.88      3080    0.361959    0.340103
2017-09-11 15:54:58.243915 min lat: 0.205299 max lat: 1.01318 avg lat: 0.341398
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60     236     44870     44634   2973.32      2232    0.268238    0.341398
Total time run:       60.291124
Total reads made:     44870
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   2976.89
Average IOPS:         744
Stddev IOPS:          107
Max IOPS:             988
Min IOPS:             540
Average Latency(s):   0.341769
Max latency(s):       1.01318
Min latency(s):       0.205299
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom