Hello, i have Proxmox Test 8.3 with Ceph Reef and 100GB Network:
With Rados Bench i get 1.2 GB write und 2.8 GB read:
Single Linux Container i get 180 MB write und 328 MB read
With 3 Container at same time i get a total of 420 MB write and 960 MB read:
Do i understand somethign wrong ? why does a signle container only get 1/6 of the speed of the Rados Pool Benchmark ?
Code:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 52.39632 root default
-3 17.46544 host pve01
0 ssd 3.49309 osd.0 up 1.00000 1.00000
1 ssd 3.49309 osd.1 up 1.00000 1.00000
6 ssd 3.49309 osd.6 up 1.00000 1.00000
9 ssd 3.49309 osd.9 up 1.00000 1.00000
12 ssd 3.49309 osd.12 up 1.00000 1.00000
-5 17.46544 host pve02
2 ssd 3.49309 osd.2 up 1.00000 1.00000
4 ssd 3.49309 osd.4 up 1.00000 1.00000
7 ssd 3.49309 osd.7 up 1.00000 1.00000
10 ssd 3.49309 osd.10 up 1.00000 1.00000
13 ssd 3.49309 osd.13 up 1.00000 1.00000
-7 17.46544 host pve03
3 ssd 3.49309 osd.3 up 1.00000 1.00000
5 ssd 3.49309 osd.5 up 1.00000 1.00000
8 ssd 3.49309 osd.8 up 1.00000 1.00000
11 ssd 3.49309 osd.11 up 1.00000 1.00000
14 ssd 3.49309 osd.14 up 1.00000 1.00000With Rados Bench i get 1.2 GB write und 2.8 GB read:
Code:
rados bench -p pool1ssd 600 write -t 16 --object_size=4MB --no-cleanup
Total time run: 600.085
Total writes made: 166749
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1111.5
Stddev Bandwidth: 44.7943
Max bandwidth (MB/sec): 1236
Min bandwidth (MB/sec): 588
Average IOPS: 277
Stddev IOPS: 11.1986
Max IOPS: 309
Min IOPS: 147
Average Latency(s): 0.0575704
Stddev Latency(s): 0.0262829
Max latency(s): 1.47922
Min latency(s): 0.0166325
rados bench -p pool1ssd 120 seq -t 16
Total time run: 120.026
Total reads made: 85319
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 2843.36
Average IOPS: 710
Stddev IOPS: 35.7975
Max IOPS: 790
Min IOPS: 632
Average Latency(s): 0.0219009
Max latency(s): 0.251782
Min latency(s): 0.0111295Single Linux Container i get 180 MB write und 328 MB read
Code:
fio --ioengine=psync --filename=/tmp/test_fio --size=20G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --rw=write --bs=4M --numjobs=1 --iodepth=16
WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=105GiB
fio --ioengine=psync --filename=/tmp/test_fio --size=20G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --rw=read --bs=4M --numjobs=1 --iodepth=16
READ: bw=326MiB/s (342MB/s), 326MiB/s-326MiB/s (342MB/s-342MB/s), io=191GiB (205GB), run=600002-600002msecWith 3 Container at same time i get a total of 420 MB write and 960 MB read:
Code:
3 Container write:
WRITE: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=78.8GiB (84.6GB), run=600014-600014msec
WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=77.3GiB (83.0GB), run=600014-600014msec
WRITE: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=78.8GiB (84.6GB), run=600014-600014msec
3 Container read:
READ: bw=306MiB/s (321MB/s), 306MiB/s-306MiB/s (321MB/s-321MB/s), io=179GiB (193GB), run=600011-600011msec
READ: bw=304MiB/s (319MB/s), 304MiB/s-304MiB/s (319MB/s-319MB/s), io=178GiB (192GB), run=600007-600007msec
READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=182GiB (195GB), run=600005-600005msecDo i understand somethign wrong ? why does a signle container only get 1/6 of the speed of the Rados Pool Benchmark ?
