[SOLVED] Ceph performance after upgrade to pacific extremely slow

G

garbled

Member
Feb 9, 2021
44
2
13
50
I'm not 100% sure how to characterize this with data, but my ceph performance has become basically awful, after my upgrade from octopus to pacific. I've never had what I would call fast performance from the cluster, but it was fine for daily use. I would get alerts from zabbix about high IO wait on a random VM or two about twice a day. I upgraded PVE from 6 to 7 about a month prior with no issues, it was not until I did the ceph upgrade this started happening.

After the upgrade, my performance is completely trashed. Every 30 minutes every VM on my network screams errors about high IO wait. My VM's are horribly slow. I used to be able to mark an OSD as out, and have it complete backfill in an hour or two. Now it takes 6-8 hours. During which time, osd's start flapping randomly and marking themselves as down, as well as screaming about slow ops. I've never seen that error once in 2 years prior. Right now my Vm backup is running, by which I mean, still running, from Friday at midnight. Prior to upgrade, it would take 5 hours.

I've done fio tests of the raw drives, they are about the same as they were before the upgrade. I tried replacing one with a new drive, and the new drive is now screaming about slow ops. In the OSD page, prior to upgrade, my SSD's would report apply/commit of an avg of 2-11 any time I ever looked. Now I commonly see numbers in the thousands. I've done iperf checks of the ceph network, and cannot find any bottlenecks, all adapters are showing full 10G performance.

To fix this, I've tried moving disks from some of my slower machines, to my faster machines. No impact. I've tried destroying an OSD and re-creating it, No impact, other than my cluster gets infinitely worse for about 15 hours while it rebuilds. (Previously this same rebuild would take maybe an hour or two max).

I'm at my wits end here. My entire cluster is basically falling apart right now, and I can't figure out anything that is causing it. None of my hardware is great, but it isn't this bad. I see that a few other users have posted issues that look identical to mine, but with no resolution:

https://forum.proxmox.com/threads/ceph-perfomance-degraded-after-upgraded-to-pve-7.109701/
https://forum.proxmox.com/threads/v...-upgrade-to-proxmox-7-and-ceph-pacific.97002/

The latter one basically matches my experience perfectly. The exact same symptoms. Other than migrating everything I have off ceph and then reinstalling octopus, what can I do here?
 
Some examples of horrible performance:

1653839521656.png

1653839609637.png


Iperf3 tests from all 5 nodes look pretty much identical:
Code: 
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.1.9, port 41392
[  5] local 10.1.1.8 port 5201 connected to 10.1.1.9 port 41394
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.07 GBytes  9.16 Gbits/sec                 
[  5]   1.00-2.00   sec  1.13 GBytes  9.74 Gbits/sec                 
[  5]   2.00-3.00   sec  1.14 GBytes  9.83 Gbits/sec                 
[  5]   3.00-4.00   sec  1.11 GBytes  9.51 Gbits/sec                 
[  5]   4.00-5.00   sec  1.14 GBytes  9.81 Gbits/sec                 
[  5]   5.00-6.00   sec  1.12 GBytes  9.59 Gbits/sec                 
[  5]   6.00-7.00   sec  1.14 GBytes  9.78 Gbits/sec                 
[  5]   7.00-8.00   sec  1.12 GBytes  9.61 Gbits/sec                 
[  5]   8.00-9.00   sec  1.15 GBytes  9.84 Gbits/sec                 
[  5]   9.00-10.00  sec  1.14 GBytes  9.81 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.3 GBytes  9.67 Gbits/sec                  receiver

I could understand if maybe 1-2 drives were getting old and making life difficult, but this is basically every drive, on all systems, all at once.
 
We had the same problem this weekend.
"ceph osd perf" was out of control.

Source of the problem seemed to be the snaptrims.

We disabled snaptrim with " ceph osd set nosnaptrim" to let the IO settle and then:

ceph tell 'osd.*' injectargs '--osd-max-trimming-pgs 1'
ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep 30'
ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep-ssd 30'
ceph tell 'osd.*' injectargs '--osd-pg-max-concurrent-snap-trims 1'
 
  • Like
Reactions: pmuch
Curious, did you see any snaptrims running? I never see any running at all. Either way, I'm thinking I'll try that.


osd.0: osd_pg_max_concurrent_snap_trims = '1' (not observed, change may require restart)

Interesting, might have to restart all osds...
 
Hrmm, seems to have not solved anything. I'm not seeing any snaptrims running.I also tried upgrading qemu-kvm, that seems to have done nothing (even after restarting or migrating the vm's around)

What I see, is just massive io load, for no reason. If I look at the VM's, none of them are really doing anything in particular. The big ones are even mostly idle at the time this happens. But then every 20 minutes or so, massive IO spike.

It's worth noting, I attempted to run fio on a VM, and it basically never completed, just hung forever. Just a read test. My *read* performance is trash. and while I might believe that all my SSD's have decided to simultaneously grow old and slow (even the brand new one), the spinning rust in the array is also complaining of slow ops.

I'm kinda wondering if it's something in the kernel... another post mentioned problems with specific kernels causing issues. Maybe I'll downgrade my kernel on one of the hosts and see if it stops screaming at me. At this point it either has to be the kernel or ceph.
 
does "ceph osd set nosnaptrim" bring your IO back up again?

in my case:

root@cluster5-node01:~# rados bench -p scbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_cluster5-node01_1969657 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 63 47 187.986 188 0.0322389 0.089592 2 16 87 71 141.987 96 0.0188486 0.0699226 3 16 90 74 98.6575 12 0.0251653 0.101937 4 16 132 116 115.989 168 0.0155427 0.299761 5 16 132 116 92.791 0 - 0.299761 6 16 172 156 103.99 80 0.0166353 0.403856 7 16 181 165 94.2764 36 0.0193426 0.385475 8 16 195 179 89.4908 56 0.0195011 0.487974 9 16 195 179 79.5469 0 - 0.487974 10 16 203 187 74.7919 16 0.0194182 0.614515 11 13 203 190 69.0832 12 3.24139 0.675844 12 13 203 190 63.3261 0 - 0.675844 13 8 203 195 59.9931 10 5.4442 0.791678 14 4 203 199 56.8503 16 4.68408 0.935441 15 4 203 199 53.0603 0 - 0.935441 16 4 203 199 49.744 0 - 0.935441 17 4 203 199 46.8178 0 - 0.935441 18 4 203 199 44.2168 0 - 0.935441 19 4 203 199 41.8896 0 - 0.935441 2022-05-30T19:40:07.033591+0200 min lat: 0.0145853 max lat: 10.508 avg lat: 0.935441 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 4 203 199 39.795 0 - 0.935441

and with ceph osd set nosnaptrim:


root@cluster5-node01:~# rados bench -p scbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_cluster5-node01_1971946 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 205 189 755.958 756 0.0635206 0.0811364 2 16 398 382 763.937 772 0.090218 0.0819544 3 16 604 588 783.934 824 0.080734 0.0802408 4 16 802 786 785.926 792 0.0861542 0.0802062 5 16 998 982 785.519 784 0.0753882 0.0805595 6 16 1195 1179 785.91 788 0.109717 0.080902 7 16 1404 1388 793.051 836 0.0778741 0.0799894 8 16 1612 1596 797.908 832 0.0829531 0.0798218 9 16 1808 1792 796.352 784 0.0821584 0.0798276 10 16 2010 1994 797.507 808 0.101579 0.0798826

is this the case for you, too?
 
No impact:

root@alphard:~# rados bench -p bench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_alphard_1575585 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 69 53 211.987 212 0.808102 0.24556 2 16 133 117 233.975 256 0.0712626 0.23791 3 16 210 194 258.636 308 0.071895 0.232147 4 16 287 271 270.966 308 0.185845 0.226517 5 16 325 309 247.167 152 0.0994524 0.221298 6 16 325 309 205.971 0 - 0.221298 7 16 325 309 176.546 0 - 0.221298 8 16 325 309 154.478 0 - 0.221298 9 16 325 309 137.314 0 - 0.221298 10 16 325 309 123.582 0 - 0.221298 11 16 325 309 112.347 0 - 0.221298 12 16 325 309 102.985 0 - 0.221298 13 16 325 309 95.0634 0 - 0.221298 14 16 325 309 88.2733 0 - 0.221298 15 16 325 309 82.3882 0 - 0.221298 16 16 325 309 77.2388 0 - 0.221298 17 16 325 309 72.6953 0 - 0.221298 18 16 325 309 68.6566 0 - 0.221298 19 16 325 309 65.043 0 - 0.221298 2022-05-30T11:44:15.813644-0700 min lat: 0.0428174 max lat: 0.998697 avg lat: 0.221298

Then set nosnaptrim, and you can see it continues it's downward spiral

root@alphard:~# rados bench -p bench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_alphard_1577778 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 37 21 83.9885 84 0.104992 0.229134 2 16 49 33 65.9915 48 0.115815 0.185393 3 16 49 33 43.9945 0 - 0.185393 4 16 49 33 32.9956 0 - 0.185393 5 16 49 33 26.3963 0 - 0.185393 6 16 49 33 21.9968 0 - 0.185393 7 16 49 33 18.8544 0 - 0.185393 8 16 49 33 16.4975 0 - 0.185393 9 16 49 33 14.6644 0 - 0.185393 10 16 49 33 13.1979 0 - 0.185393 11 16 49 33 11.9981 0 - 0.185393 12 16 49 33 10.9983 0 - 0.185393 13 16 49 33 10.1523 0 - 0.185393 14 16 49 33 9.4271 0 - 0.185393 15 16 49 33 8.79865 0 - 0.185393 16 16 49 33 8.24872 0 - 0.185393 17 16 49 33 7.76352 0 - 0.185393 18 16 49 33 7.33221 0 - 0.185393 19 16 49 33 6.94632 0 - 0.185393 2022-05-30T11:46:19.091601-0700 min lat: 0.051549 max lat: 0.471077 avg lat: 0.185393

one more try..
root@alphard:~# rados bench -p bench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_alphard_1578699 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 46 30 119.987 120 0.602295 0.232467 2 16 79 63 125.981 132 0.0587483 0.201476 3 16 80 64 85.3204 4 0.0679337 0.19939 4 16 80 64 63.9898 0 - 0.19939 5 16 80 64 51.1917 0 - 0.19939 6 16 80 64 42.6597 0 - 0.19939 7 16 80 64 36.5656 0 - 0.19939 8 16 80 64 31.9951 0 - 0.19939 9 16 80 64 28.4401 0 - 0.19939 10 16 80 64 25.5962 0 - 0.19939 11 16 80 64 23.2692 0 - 0.19939


osd commit_latency(ms) apply_latency(ms) 12 3 3 11 2121 2121 10 55 55 9 29 29 8 35 35 7 18 18 6 12 12 5 3 3 2 5 5 14 3 3 1 27 27 13 3 3 0 5 5

Basically, the longer I run the bench, the worse it gets.. it gets down to like 7 or so eventually
 
Last edited:
Full runs below. It's horrific.

root@alphard:~# rados bench -p bench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_alphard_1594585 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 23 7 27.9968 28 0.210996 0.20898 2 16 23 7 13.9983 0 - 0.20898 3 16 23 7 9.33211 0 - 0.20898 4 16 23 7 6.99905 0 - 0.20898 5 16 23 7 5.59923 0 - 0.20898 6 16 23 7 4.66599 0 - 0.20898 7 16 23 7 3.99941 0 - 0.20898 8 16 23 7 3.49949 0 - 0.20898 9 16 23 7 3.11065 0 - 0.20898 10 16 25 9 3.59947 0.888889 9.89483 2.32414 11 16 25 9 3.27225 0 - 2.32414 12 16 25 9 2.99957 0 - 2.32414 13 14 25 11 3.38412 2.66667 12.4626 4.16734 14 14 25 11 3.1424 0 - 4.16734 15 14 25 11 2.93291 0 - 4.16734 16 11 25 14 3.49949 4 15.6113 6.56992 17 2 25 23 5.41097 36 16.4762 10.0163 18 2 25 23 5.11037 0 - 10.0163 19 1 25 24 5.0519 2 8.61294 9.95785 2022-05-30T12:02:40.812198-0700 min lat: 0.132283 max lat: 16.6373 avg lat: 9.95785 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 1 25 24 4.7993 0 - 9.95785 21 1 25 24 4.57076 0 - 9.95785 22 1 25 24 4.36299 0 - 9.95785 23 1 25 24 4.1733 0 - 9.95785 Total time run: 23.3231 Total writes made: 25 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 4.28759 Stddev Bandwidth: 9.22611 Max bandwidth (MB/sec): 36 Min bandwidth (MB/sec): 0 Average IOPS: 1 Stddev IOPS: 2.32022 Max IOPS: 9 Min IOPS: 0 Average Latency(s): 10.4924 Stddev Latency(s): 7.33597 Max latency(s): 23.3226 Min latency(s): 0.132283 root@alphard:~# rados bench -p bench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 6 25 19 75.9804 76 0.109395 0.0575924 2 6 25 19 37.993 0 - 0.0575924 3 6 25 19 25.3293 0 - 0.0575924 4 6 25 19 18.9973 0 - 0.0575924 5 6 25 19 15.198 0 - 0.0575924 6 6 25 19 12.665 0 - 0.0575924 7 6 25 19 10.8557 0 - 0.0575924 8 6 25 19 9.4987 0 - 0.0575924 9 6 25 19 8.44328 0 - 0.0575924 10 6 25 19 7.59896 0 - 0.0575924 11 6 25 19 6.90814 0 - 0.0575924 12 6 25 19 6.33246 0 - 0.0575924 13 5 25 20 6.153 0.333333 12.6432 0.686875 14 5 25 20 5.71351 0 - 0.686875 15 5 25 20 5.33262 0 - 0.686875 16 2 25 23 5.74923 4 15.8559 2.62265 17 2 25 23 5.41104 0 - 2.62265 18 2 25 23 5.11042 0 - 2.62265 19 2 25 23 4.84144 0 - 2.62265 2022-05-30T12:03:20.130712-0700 min lat: 0.0219833 max lat: 15.8559 avg lat: 2.62265 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 2 25 23 4.59937 0 - 2.62265 21 2 25 23 4.38034 0 - 2.62265 22 2 25 23 4.18122 0 - 2.62265 23 2 25 23 3.99944 0 - 2.62265 24 2 25 23 3.8328 0 - 2.62265 25 2 25 23 3.67949 0 - 2.62265 26 2 25 23 3.53797 0 - 2.62265 27 2 25 23 3.40694 0 - 2.62265 28 2 25 23 3.28527 0 - 2.62265 29 2 25 23 3.17198 0 - 2.62265 30 2 25 23 3.06625 0 - 2.62265 31 2 25 23 2.96733 0 - 2.62265 32 2 25 23 2.8746 0 - 2.62265 33 2 25 23 2.78749 0 - 2.62265 34 2 25 23 2.70551 0 - 2.62265 Total time run: 34.5388 Total reads made: 25 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 2.89529 Average IOPS: 0 Stddev IOPS: 3.25779 Max IOPS: 19 Min IOPS: 0 Average Latency(s): 5.1579 Max latency(s): 34.5374 Min latency(s): 0.0219833 root@alphard:~# rados bench -p bench 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 90 74 295.93 296 0.00883993 0.0381576 2 16 90 74 147.974 0 - 0.0381576 3 16 116 100 133.313 52 0.00447025 0.115956 4 16 117 101 100.986 4 3.93399 0.153758 5 16 122 106 84.7882 20 0.0324745 0.184669 6 16 122 106 70.6565 0 - 0.184669 7 16 128 112 63.9908 12 0.00880388 0.235949 8 16 128 112 55.9918 0 - 0.235949 9 16 200 184 81.7662 144 0.0190615 0.776235 10 15 461 446 178.374 1048 0.0710378 0.356196 Total time run: 10.0808 Total reads made: 461 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 182.921 Average IOPS: 45 Stddev IOPS: 81.7152 Max IOPS: 262 Min IOPS: 0 Average Latency(s): 0.346741 Max latency(s): 8.56759 Min latency(s): 0.00447025 root@alphard:~# time rados -p bench cleanup Removed 25 objects real 0m12.663s user 0m0.050s sys 0m0.009s root@alphard:~#
 
I discovered that the CPU load for the OSDs at 100%. I would guess duo to very many small iops. Bit iostat does not show many iops.

Anyway...so i ended up deleting ALL snapshots after i did not get a reply here:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F47LN362QSMB2ZCY5QFD2QZJTD2KY2BO/

after deleting ALL snapshots, and restarting all OSDs i am back to normal performance.

Short steps:
  1. set nosnaptrim, noout
  2. delete all snapshots
  3. restart all OSDs
  4. remove noout
After that i do not have any snaptrim objects/pgs anymore.

The CPU Load of the OSDs dropped suddenly on the whole cluster after restarting a few specific ODSs.
My Plan is now to free some space (since i am almost full) and then recreate those OSDs.

Maybe that helps you, too.
 
OK, interesting.. I normally don't keep snapshots around, but I found one vm, which is semi-active, that had a really old snapshot.. I've just told it to delete, and now I see a ton of snaptrims running. Maybe I'll let those go and see what happens? I've never really seen a snaptrim run before on this cluster.
 
I might be on to something.. on saturday or sunday, I upgraded pve-qemu-kvm, because I saw something in another thread about it causing issues with ceph and backups. After doing so, I moved 2 of my heaviest use VM's back and forth between nodes, and it looks like the massive slowdown has somewhat abated. I'm still seeing issues, but not nearly as severe. Right now I'm trying to bulk migrate all my stuff around, and will see if that does anything to fix it up.
 
So, yes, 100%, I think I've figured it out.

There were two problems. The one that woke me up and freaked me out was this one: (Month average)

1654228080409.png

The 14th was the day I upgraded ceph. However, I also did a general update on that day, and got:
2022-05-14 06:45:15 status installed pve-qemu-kvm:amd64 6.2.0-6

And the fix?

2022-05-30 07:20:58 upgrade pve-qemu-kvm:amd64 6.2.0-6 6.2.0-8
(followed by bounce or migrate of all VM's)


The second problem is more sinister, and made debugging the first super hard. My SSD's (Crucial BX500's) as they age have slowly gotten slower and twitchier. Which causes random massive spikes in latency (in the thousands), leading to this: (Hour average)


1654228320512.png

This is still ongoing, but it's fixable, and my fault. This one is minor and spiky, and I can live with it, my VM's aren't all exploding left and right simultaneously permanently. This one is just annoying and periodic. (For example, this is still happening, but you don't even see it in the monthly graph, it's probably been happening all along. The monthly one was just everything broken all the time.)
 
Hi,
i take advantage of the post for a similar problem. I don't have IO delay problems, but my VMs can't go over 100Mb/s. I updated everything tonight and did all the bech on ceph. I can't find the bottleneck.
Maybe someone can see something that has escaped me.


Code: 
root@cl1-hv1:~# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_cl1-hv1_705763
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15       268       253   1011.94      1012   0.0725292    0.054169
    2      16       544       528   1055.91      1100   0.0465715   0.0482364
    3      16       788       772   1029.24       976     2.66004   0.0542352
    4      16      1076      1060    1059.9      1152   0.0519425   0.0600233
    5      16      1361      1345    1075.9      1140   0.0428953   0.0572348
    6      16      1632      1616   1077.23      1084    0.053529    0.054498
    7      16      1855      1839   1050.76       892     2.42942   0.0574864
    8      16      2118      2102    1050.9      1052   0.0418309   0.0605645
    9      16      2384      2368   1052.34      1064   0.0223319   0.0597881
   10      16      2659      2643    1057.1      1100   0.0509571   0.0583913
Total time run:         10.9996
Total writes made:      2659
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     966.944
Stddev Bandwidth:       79.0398
Max bandwidth (MB/sec): 1152
Min bandwidth (MB/sec): 892
Average IOPS:           241
Stddev IOPS:            19.7599
Max IOPS:               288
Min IOPS:               223
Average Latency(s):     0.0620876
Stddev Latency(s):      0.180811
Max latency(s):         2.96745
Min latency(s):         0.0125057


root@cl1-hv1:~# rados bench -p testbench 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       436       420   1679.58      1680   0.0453072   0.0365082
    2      16       761       745   1489.71      1300   0.0125278   0.0414372
    3      16      1127      1111   1481.06      1464    0.162533   0.0417105
    4      16      1546      1530   1529.75      1676   0.0125646   0.0406801
    5      15      1917      1902   1521.13      1488   0.0146329   0.0409687
    6      16      2318      2302   1534.25      1600   0.0131746   0.0406136
Total time run:       6.84313
Total reads made:     2659
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1554.26
Average IOPS:         388
Stddev IOPS:          36.6752
Max IOPS:             420
Min IOPS:             325
Average Latency(s):   0.0401443
Max latency(s):       0.27127
Min latency(s):       0.00740274


root@cl1-hv1:~# rados bench -p testbench 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       480       464   1855.65      1856   0.0153482   0.0328741
    2      16       948       932   1863.72      1872   0.0248183   0.0331097
    3      15      1414      1399   1864.96      1868   0.0265215   0.0331663
    4      16      1857      1841   1840.68      1768   0.0100894   0.0336224
    5      16      2323      2307    1845.3      1864   0.0681191   0.0336888
    6      15      2774      2759   1838.91      1808   0.0394973   0.0338325
    7      16      3237      3221   1840.18      1848   0.0235172   0.0337906
    8      15      3673      3658   1828.24      1748   0.0150063   0.0340481
    9      16      4148      4132   1835.74      1896   0.0424218   0.0339623
   10      13      4606      4593   1836.33      1844  0.00904082   0.0339439
Total time run:       10.0323
Total reads made:     4606
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1836.47
Average IOPS:         459
Stddev IOPS:          11.9168
Max IOPS:             474
Min IOPS:             437
Average Latency(s):   0.0339434
Max latency(s):       0.176696
Min latency(s):       0.00472854




root@cl1-hv1:~# ceph tell osd.* bench -f plain
osd.0: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40906894199999999,
    "bytes_per_sec": 2624843183.523818,
    "iops": 625.81138218017054
}
osd.1: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41473186099999998,
    "bytes_per_sec": 2589002497.6885004,
    "iops": 617.26629678928862
}
osd.2: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.45614067800000002,
    "bytes_per_sec": 2353970772.148499,
    "iops": 561.23036674225307
}
osd.3: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.37090004900000001,
    "bytes_per_sec": 2894962745.0709772,
    "iops": 690.21290423178129
}
osd.4: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.43318344600000003,
    "bytes_per_sec": 2478723122.7667918,
    "iops": 590.97364491624637
}
osd.5: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40720621299999998,
    "bytes_per_sec": 2636850297.7630157,
    "iops": 628.6741012961902
}
osd.6: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.39124514700000002,
    "bytes_per_sec": 2744422089.9179611,
    "iops": 654.32121513318089
}
osd.7: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.39964849000000002,
    "bytes_per_sec": 2686715578.482481,
    "iops": 640.56291067182565
}
osd.8: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.42292355700000001,
    "bytes_per_sec": 2538855559.6585035,
    "iops": 605.31033507788266
}
osd.9: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.402220048,
    "bytes_per_sec": 2669538302.078866,
    "iops": 636.46752883884096
}
osd.10: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.409613483,
    "bytes_per_sec": 2621353711.6403952,
    "iops": 624.97942725191001
}
osd.11: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.38223692199999998,
    "bytes_per_sec": 2809100226.063457,
    "iops": 669.74168445192743
}
osd.12: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41980739299999997,
    "bytes_per_sec": 2557701083.6490917,
    "iops": 609.8034581301431
}
osd.13: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40532001499999998,
    "bytes_per_sec": 2649121149.3713183,
    "iops": 631.59970030100783
}
osd.14: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.43137950600000002,
    "bytes_per_sec": 2489088630.9281459,
    "iops": 593.44497464374206
}
osd.15: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.38872874200000002,
    "bytes_per_sec": 2762187890.907228,
    "iops": 658.55691216164303
}
osd.16: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41295932800000001,
    "bytes_per_sec": 2600115195.848052,
    "iops": 619.91577049447346
}
osd.17: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.45141120099999998,
    "bytes_per_sec": 2378633542.1481934,
    "iops": 567.11042932228884
}
osd.18: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.42199397700000002,
    "bytes_per_sec": 2544448220.88065,
    "iops": 606.64372941986323
}
osd.19: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.37785223200000001,
    "bytes_per_sec": 2841697714.2535443,
    "iops": 677.51353126848801
}
osd.20: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41133299400000001,
    "bytes_per_sec": 2610395566.7606864,
    "iops": 622.36680192010078
}
osd.21: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.44328449199999997,
    "bytes_per_sec": 2422240893.5523963,
    "iops": 577.50723208246143
}
osd.22: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.484685598,
    "bytes_per_sec": 2215336763.5239701,
    "iops": 528.17744339083913
}
osd.23: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41286162999999998,
    "bytes_per_sec": 2600730477.1819072,
    "iops": 620.06246499583892
}
osd.24: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.44045880199999998,
    "bytes_per_sec": 2437780376.1088195,
    "iops": 581.21213343353736
}
osd.25: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.47458924000000002,
    "bytes_per_sec": 2262465588.1368065,
    "iops": 539.41383078975832
}
osd.26: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.51593041799999995,
    "bytes_per_sec": 2081175651.8686211,
    "iops": 496.19094177928474
}
osd.27: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41869582999999999,
    "bytes_per_sec": 2564491325.3614206,
    "iops": 611.42237791095272
}
osd.28: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41918551500000001,
    "bytes_per_sec": 2561495532.5925326,
    "iops": 610.70812525571171
}
osd.29: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.415986725,
    "bytes_per_sec": 2581192522.4296522,
    "iops": 615.40425358525567
}



root@cl1-hv1:~# iperf -c xxx.xxx.xxx.xxx
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 50986 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0006 sec  11.3 GBytes  9.67 Gbits/sec




[root@vm ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=76.1MiB/s,w=25.3MiB/s][r=19.5k,w=6471 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3567774: Sat Jun 25 10:06:24 2022
  read: IOPS=20.7k, BW=80.8MiB/s (84.7MB/s)(3070MiB/37991msec)
   bw (  KiB/s): min= 8105, max=101232, per=100.00%, avg=83377.01, stdev=22802.14, samples=75
   iops        : min= 2026, max=25308, avg=20844.11, stdev=5700.62, samples=75
  write: IOPS=6913, BW=27.0MiB/s (28.3MB/s)(1026MiB/37991msec); 0 zone resets
   bw (  KiB/s): min= 2398, max=33808, per=100.00%, avg=27866.80, stdev=7642.36, samples=75
   iops        : min=  599, max= 8452, avg=6966.56, stdev=1910.68, samples=75
  cpu          : usr=6.45%, sys=28.74%, ctx=375645, majf=0, minf=143
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=80.8MiB/s (84.7MB/s), 80.8MiB/s-80.8MiB/s (84.7MB/s-84.7MB/s), io=3070MiB (3219MB), run=37991-37991msec
  WRITE: bw=27.0MiB/s (28.3MB/s), 27.0MiB/s-27.0MiB/s (28.3MB/s-28.3MB/s), io=1026MiB (1076MB), run=37991-37991msec

Disk stats (read/write):
    dm-0: ios=788943/289954, merge=0/0, ticks=1012902/1657872, in_queue=2670774, util=99.51%, aggrios=788932/267760, aggrmerge=14/23059, aggrticks=1011251/1378083, aggrin_queue=1833260, aggrutil=99.46%
  sda: ios=788932/267760, merge=14/23059, ticks=1011251/1378083, in_queue=1833260, util=99.46%
 
Last edited:
Sean510 said:
Hi,
i take advantage of the post for a similar problem. I don't have IO delay problems, but my VMs can't go over 100Mb/s. I updated everything tonight and did all the bech on ceph. I can't find the bottleneck.
Maybe someone can see something that has escaped me.


Code: 
root@cl1-hv1:~# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_cl1-hv1_705763
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15       268       253   1011.94      1012   0.0725292    0.054169
    2      16       544       528   1055.91      1100   0.0465715   0.0482364
    3      16       788       772   1029.24       976     2.66004   0.0542352
    4      16      1076      1060    1059.9      1152   0.0519425   0.0600233
    5      16      1361      1345    1075.9      1140   0.0428953   0.0572348
    6      16      1632      1616   1077.23      1084    0.053529    0.054498
    7      16      1855      1839   1050.76       892     2.42942   0.0574864
    8      16      2118      2102    1050.9      1052   0.0418309   0.0605645
    9      16      2384      2368   1052.34      1064   0.0223319   0.0597881
   10      16      2659      2643    1057.1      1100   0.0509571   0.0583913
Total time run:         10.9996
Total writes made:      2659
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     966.944
Stddev Bandwidth:       79.0398
Max bandwidth (MB/sec): 1152
Min bandwidth (MB/sec): 892
Average IOPS:           241
Stddev IOPS:            19.7599
Max IOPS:               288
Min IOPS:               223
Average Latency(s):     0.0620876
Stddev Latency(s):      0.180811
Max latency(s):         2.96745
Min latency(s):         0.0125057


root@cl1-hv1:~# rados bench -p testbench 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       436       420   1679.58      1680   0.0453072   0.0365082
    2      16       761       745   1489.71      1300   0.0125278   0.0414372
    3      16      1127      1111   1481.06      1464    0.162533   0.0417105
    4      16      1546      1530   1529.75      1676   0.0125646   0.0406801
    5      15      1917      1902   1521.13      1488   0.0146329   0.0409687
    6      16      2318      2302   1534.25      1600   0.0131746   0.0406136
Total time run:       6.84313
Total reads made:     2659
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1554.26
Average IOPS:         388
Stddev IOPS:          36.6752
Max IOPS:             420
Min IOPS:             325
Average Latency(s):   0.0401443
Max latency(s):       0.27127
Min latency(s):       0.00740274


root@cl1-hv1:~# rados bench -p testbench 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       480       464   1855.65      1856   0.0153482   0.0328741
    2      16       948       932   1863.72      1872   0.0248183   0.0331097
    3      15      1414      1399   1864.96      1868   0.0265215   0.0331663
    4      16      1857      1841   1840.68      1768   0.0100894   0.0336224
    5      16      2323      2307    1845.3      1864   0.0681191   0.0336888
    6      15      2774      2759   1838.91      1808   0.0394973   0.0338325
    7      16      3237      3221   1840.18      1848   0.0235172   0.0337906
    8      15      3673      3658   1828.24      1748   0.0150063   0.0340481
    9      16      4148      4132   1835.74      1896   0.0424218   0.0339623
   10      13      4606      4593   1836.33      1844  0.00904082   0.0339439
Total time run:       10.0323
Total reads made:     4606
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1836.47
Average IOPS:         459
Stddev IOPS:          11.9168
Max IOPS:             474
Min IOPS:             437
Average Latency(s):   0.0339434
Max latency(s):       0.176696
Min latency(s):       0.00472854




root@cl1-hv1:~# ceph tell osd.* bench -f plain
osd.0: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40906894199999999,
    "bytes_per_sec": 2624843183.523818,
    "iops": 625.81138218017054
}
osd.1: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41473186099999998,
    "bytes_per_sec": 2589002497.6885004,
    "iops": 617.26629678928862
}
osd.2: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.45614067800000002,
    "bytes_per_sec": 2353970772.148499,
    "iops": 561.23036674225307
}
osd.3: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.37090004900000001,
    "bytes_per_sec": 2894962745.0709772,
    "iops": 690.21290423178129
}
osd.4: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.43318344600000003,
    "bytes_per_sec": 2478723122.7667918,
    "iops": 590.97364491624637
}
osd.5: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40720621299999998,
    "bytes_per_sec": 2636850297.7630157,
    "iops": 628.6741012961902
}
osd.6: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.39124514700000002,
    "bytes_per_sec": 2744422089.9179611,
    "iops": 654.32121513318089
}
osd.7: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.39964849000000002,
    "bytes_per_sec": 2686715578.482481,
    "iops": 640.56291067182565
}
osd.8: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.42292355700000001,
    "bytes_per_sec": 2538855559.6585035,
    "iops": 605.31033507788266
}
osd.9: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.402220048,
    "bytes_per_sec": 2669538302.078866,
    "iops": 636.46752883884096
}
osd.10: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.409613483,
    "bytes_per_sec": 2621353711.6403952,
    "iops": 624.97942725191001
}
osd.11: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.38223692199999998,
    "bytes_per_sec": 2809100226.063457,
    "iops": 669.74168445192743
}
osd.12: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41980739299999997,
    "bytes_per_sec": 2557701083.6490917,
    "iops": 609.8034581301431
}
osd.13: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.40532001499999998,
    "bytes_per_sec": 2649121149.3713183,
    "iops": 631.59970030100783
}
osd.14: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.43137950600000002,
    "bytes_per_sec": 2489088630.9281459,
    "iops": 593.44497464374206
}
osd.15: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.38872874200000002,
    "bytes_per_sec": 2762187890.907228,
    "iops": 658.55691216164303
}
osd.16: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41295932800000001,
    "bytes_per_sec": 2600115195.848052,
    "iops": 619.91577049447346
}
osd.17: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.45141120099999998,
    "bytes_per_sec": 2378633542.1481934,
    "iops": 567.11042932228884
}
osd.18: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.42199397700000002,
    "bytes_per_sec": 2544448220.88065,
    "iops": 606.64372941986323
}
osd.19: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.37785223200000001,
    "bytes_per_sec": 2841697714.2535443,
    "iops": 677.51353126848801
}
osd.20: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41133299400000001,
    "bytes_per_sec": 2610395566.7606864,
    "iops": 622.36680192010078
}
osd.21: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.44328449199999997,
    "bytes_per_sec": 2422240893.5523963,
    "iops": 577.50723208246143
}
osd.22: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.484685598,
    "bytes_per_sec": 2215336763.5239701,
    "iops": 528.17744339083913
}
osd.23: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41286162999999998,
    "bytes_per_sec": 2600730477.1819072,
    "iops": 620.06246499583892
}
osd.24: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.44045880199999998,
    "bytes_per_sec": 2437780376.1088195,
    "iops": 581.21213343353736
}
osd.25: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.47458924000000002,
    "bytes_per_sec": 2262465588.1368065,
    "iops": 539.41383078975832
}
osd.26: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.51593041799999995,
    "bytes_per_sec": 2081175651.8686211,
    "iops": 496.19094177928474
}
osd.27: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41869582999999999,
    "bytes_per_sec": 2564491325.3614206,
    "iops": 611.42237791095272
}
osd.28: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.41918551500000001,
    "bytes_per_sec": 2561495532.5925326,
    "iops": 610.70812525571171
}
osd.29: {
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 0.415986725,
    "bytes_per_sec": 2581192522.4296522,
    "iops": 615.40425358525567
}



root@cl1-hv1:~# iperf -c xxx.xxx.xxx.xxx
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 50986 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.0000-10.0006 sec  11.3 GBytes  9.67 Gbits/sec




[root@vm ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=76.1MiB/s,w=25.3MiB/s][r=19.5k,w=6471 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3567774: Sat Jun 25 10:06:24 2022
  read: IOPS=20.7k, BW=80.8MiB/s (84.7MB/s)(3070MiB/37991msec)
   bw (  KiB/s): min= 8105, max=101232, per=100.00%, avg=83377.01, stdev=22802.14, samples=75
   iops        : min= 2026, max=25308, avg=20844.11, stdev=5700.62, samples=75
  write: IOPS=6913, BW=27.0MiB/s (28.3MB/s)(1026MiB/37991msec); 0 zone resets
   bw (  KiB/s): min= 2398, max=33808, per=100.00%, avg=27866.80, stdev=7642.36, samples=75
   iops        : min=  599, max= 8452, avg=6966.56, stdev=1910.68, samples=75
  cpu          : usr=6.45%, sys=28.74%, ctx=375645, majf=0, minf=143
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=80.8MiB/s (84.7MB/s), 80.8MiB/s-80.8MiB/s (84.7MB/s-84.7MB/s), io=3070MiB (3219MB), run=37991-37991msec
  WRITE: bw=27.0MiB/s (28.3MB/s), 27.0MiB/s-27.0MiB/s (28.3MB/s-28.3MB/s), io=1026MiB (1076MB), run=37991-37991msec

Disk stats (read/write):
    dm-0: ios=788943/289954, merge=0/0, ticks=1012902/1657872, in_queue=2670774, util=99.51%, aggrios=788932/267760, aggrmerge=14/23059, aggrticks=1011251/1378083, aggrin_queue=1833260, aggrutil=99.46%
  sda: ios=788932/267760, merge=14/23059, ticks=1011251/1378083, in_queue=1833260, util=99.46%
Click to expand...

Yes. I have the same problem. In fact, you can test RBD, if the CPU performance enough, it should be able to run full bandwidth. But this is limited to RBD testing, which would be a big discount in a virtual machine.

https://forum.proxmox.com/threads/bad-rand-read-write-i-o-proxmox-ceph.68404/#post-529486
 
You must log in or register to reply here.
Top Bottom