Poor WinServer 2022/2019 performance on Proxmox 8.2.4 / Ceph 18.2.2

J

JP29

New Member
Apr 9, 2024
5
1
3
Hi All,

I have a 6 x Dell R740xd Cluster with Proxmox 8.2.4 / Ceph 18.2.2, 26 x OSD (a mix of Toshiba and Intel enterprise-grade SSD 1.92TB drives), and a dedicated 10G network (4 x uplinks in a bond) for both the public and cluster networks, rbd pool replica x3 and EC: 5+1

Question:

We've started creating new WinServer 2022/2019 servers and have noticed it is extremely slow to work with, installing the OS takes a lot and once you are inside the VM it is very laggy.

Here is the VM config:

Code: 
# qm config 115
agent: 1
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 5
cpu: host
efidisk0: DFXSCErasureStorage:vm-115-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: ISO-Templates:iso/virtio-win-0.1.248.iso,media=cdrom,size=715188K
ide2: ISO-Templates:iso/Win_Server_STD_CORE_2022_2108.36_X23-80100.ISO,media=cdrom,size=5827918K
machine: pc-q35-9.0
memory: 16384
meta: creation-qemu=9.0.0,ctime=1729083494
name: DFX.CE.UT.US.VM.NAPPS.TST.ROZ001.00001
net0: virtio=BC:24:11:1E:72:6C,bridge=ROZ001,firewall=1
numa: 0
ostype: win11
scsi0: DFXSCErasureStorage:vm-115-disk-1,cache=writeback,discard=on,size=60G
scsihw: virtio-scsi-pci
smbios1: uuid=1e9a06f3-e68d-472b-a104-0aced268e32c
sockets: 2
tpmstate0: DFXSCErasureStorage:vm-115-disk-2,size=4M,version=v2.0
vmgenid: a0b30c55-3760-4598-9ed6-d02b7d669441

The performance in the Replicated x 3 pool is also terrible.

I have tested deploying the VMs onto the local-lvm and the performance is good.

We've done iperf tests ( CEPH is on MTU 9000) and results are as expected:

Code: 
# iperf -c 172.21.8.22 -e -i 1
------------------------------------------------------------
Client connecting to 172.21.8.22, TCP port 5001 with pid 1431026 (1 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 172.21.8.21%ceph port 50206 connected with 172.21.8.22 port 5001 (sock=3) (icwnd/mss/irtt=87/8948/254) (ct=0.33 ms) on 2024-10-29 12:26:09 (MDT)
[ ID] Interval            Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT(var)        NetPwr
[  1] 0.0000-1.0000 sec  1.15 GBytes  9.88 Gbits/sec  9418/0         22     1773K/1081(44) us  1141939
[  1] 1.0000-2.0000 sec  1.15 GBytes  9.88 Gbits/sec  9424/0          0     1773K/1214(12) us  1017481
[  1] 2.0000-3.0000 sec  1.15 GBytes  9.88 Gbits/sec  9421/0         14     1616K/1105(26) us  1117493
[  1] 3.0000-4.0000 sec  1.15 GBytes  9.88 Gbits/sec  9425/0          4     1756K/1274(26) us  969665
[  1] 4.0000-5.0000 sec  1.15 GBytes  9.89 Gbits/sec  9429/0         18     1555K/1061(33) us  1164824
[  1] 5.0000-6.0000 sec  1.15 GBytes  9.88 Gbits/sec  9424/0          8     1747K/984(36) us  1255307
[  1] 6.0000-7.0000 sec  1.15 GBytes  9.90 Gbits/sec  9439/0          0     1747K/963(47) us  1284723
[  1] 7.0000-8.0000 sec  1.15 GBytes  9.88 Gbits/sec  9425/0          0     1747K/1271(32) us  971954
[  1] 8.0000-9.0000 sec  1.15 GBytes  9.88 Gbits/sec  9424/0          5     1669K/1156(52) us  1068532
[  1] 9.0000-10.0000 sec  1.15 GBytes  9.88 Gbits/sec  9426/0          4     1747K/993(51) us  1244194
[  1] 0.0000-10.0165 sec  11.5 GBytes  9.87 Gbits/sec  94257/0         75     1747K/953(56) us  1294234

I've followed this guide to test rdb and I don't see anything unusual, ( I might be wrong but you guys can correct me)

https://docs.redhat.com/en/document...ark#benchmarking-ceph-block-performance_admin


Code: 
~# ceph osd pool create testbench 100 100
pool 'testbench' created

~# rados bench -p testbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_DFX-VH-UT-US-VHOST0001_1431976
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       162       146   583.854       584   0.0428713   0.0871647
    2      16       307       291   581.872       580   0.0404318    0.104417
    3      16       490       474   631.875       732   0.0289956    0.093098
    4      16       630       614   613.878       560   0.0407406   0.0884081
    5      16       709       693    554.29       316   0.0440478   0.0855576
    6      16       806       790   526.561       388    0.036311   0.0813596
    7      16       965       949   542.176       636   0.0533337    0.115398
    8      16      1123      1107   553.385       632   0.0285588    0.107204
    9      16      1217      1201   533.667       376   0.0357533    0.103373
   10      16      1276      1260   503.896       236    0.169554    0.100711
   11      14      1276      1262   458.814         8   0.0396088    0.100616
   12      14      1276      1262    420.58         0           -    0.100616
   13      14      1276      1262   388.228         0           -    0.100616
   14      14      1276      1262   360.498         0           -    0.100616
   15      14      1276      1262   336.465         0           -    0.100616
Total time run:         15.1013
Total writes made:      1276
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     337.985
Stddev Bandwidth:       277.684
Max bandwidth (MB/sec): 732
Min bandwidth (MB/sec): 0
Average IOPS:           84
Stddev IOPS:            69.4209
Max IOPS:               183
Min IOPS:               0
Average Latency(s):     0.181353
Stddev Latency(s):      0.824122
Max latency(s):         8.89345
Min latency(s):         0.0274122

~# rados bench -p testbench 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       339       323   1291.57      1292   0.0423749   0.0464794
    2      16       654       638    1275.7      1260   0.0396979   0.0396754
    3      16       987       971   1294.38      1332   0.0914942   0.0481208
Total time run:       3.94605
Total reads made:     1276
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1293.45
Average IOPS:         323
Stddev IOPS:          9.0185
Max IOPS:             333
Min IOPS:             315
Average Latency(s):   0.04759
Max latency(s):       1.25676
Min latency(s):       0.0138378

rados bench -p testbench 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15       345       330    1319.3      1320   0.0220504   0.0458886
    2      16       682       666   1331.51      1344   0.0484838   0.0460847
    3      15      1022      1007   1342.18      1364   0.0657176   0.0463005
    4      16      1363      1347   1346.58      1360   0.0272982   0.0463732
    5      16      1711      1695   1355.61      1392    0.024731   0.0458902
    6      16      2060      2044    1362.3      1396   0.0249081   0.0459957
    7      16      2404      2388   1364.22      1376   0.0217204   0.0458798
    8      16      2740      2724   1361.66      1344   0.0646662   0.0459278
    9      16      3093      3077   1367.22      1412    0.026175   0.0458007
   10      14      3445      3431   1372.07      1416   0.0401794   0.0456587
Total time run:       10.0358
Total reads made:     3445
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1373.08
Average IOPS:         343
Stddev IOPS:          7.9085
Max IOPS:             354
Min IOPS:             330
Average Latency(s):   0.045714
Max latency(s):       0.353993
Min latency(s):       0.00458857

rados bench -p testbench 10 write -t 4 --run-name client1
hints = 1
Maintaining 4 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_DFX-VH-UT-US-VHOST0001_1433239
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1       4        66        62   247.969       248    0.043376   0.0457508
    2       4       123       119   237.962       228   0.0537846   0.0655272
    3       4       135       131   174.637        48   0.0417172   0.0630803
    4       4       187       183   182.966       208   0.0413069   0.0817256
    5       4       254       250   199.962       268   0.0373117    0.079732
    6       4       320       316   210.627       264   0.0535278   0.0758297
    7       4       366       362   206.819       184   0.0402602   0.0771097
    8       4       449       445   222.459       332   0.0395271   0.0709675
    9       4       456       452   200.852        28     1.20324   0.0733661
   10       4       520       516   206.361       256    0.034081   0.0771441
Total time run:         10.1
Total writes made:      520
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     205.942
Stddev Bandwidth:       97.1633
Max bandwidth (MB/sec): 332
Min bandwidth (MB/sec): 28
Average IOPS:           51
Stddev IOPS:            24.2908
Max IOPS:               83
Min IOPS:               7
Average Latency(s):     0.0773108
Stddev Latency(s):      0.156676
Max latency(s):         1.35768
Min latency(s):         0.0264381
Cleaning up (deleting benchmark objects)
Removed 520 objects
Clean up completed and total clean up time :0.800697

I also did the block performance:

Code: 
~# rbd bench --io-type write image01 --pool=testbench
bench  type write io_size 4096 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1    101360    101780   398 MiB/s
    2    182256   91315.4   357 MiB/s
    3    254448   84931.6   332 MiB/s
elapsed: 3   ops: 262144   ops/sec: 84017.6   bytes/sec: 328 MiB/s


Hope you guys can help me identify where the issue could be,

Regards

JP
 
Last edited:
Hello @wuwzy

Thanks for the response

I break down the output in two replies since the message is too large

Here is the output hosts 1-3:

Bash: 
~# ceph osd df tree
ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE   DATA     OMAP     META      AVAIL    %USE  VAR   PGS  STATUS  TYPE NAME
 -1         273.16687         -  273 TiB   6.6 TiB  6.4 TiB  1.3 MiB   209 GiB  267 TiB  2.42  1.00    -          root default
 -3          45.64432         -   46 TiB  1018 GiB  994 GiB  204 KiB    23 GiB   45 TiB  2.18  0.90    -              host DFX-VH-UT-US-VHOST0001
  0    ssd    1.74660   1.00000  1.7 TiB    39 GiB   38 GiB    8 KiB   1.3 GiB  1.7 TiB  2.21  0.91    5      up          osd.0
  1    ssd    1.74660   1.00000  1.7 TiB    52 GiB   51 GiB    7 KiB   878 MiB  1.7 TiB  2.89  1.19    5      up          osd.1
  2    ssd    1.74660   1.00000  1.7 TiB    32 GiB   31 GiB    6 KiB   267 MiB  1.7 TiB  1.78  0.73    4      up          osd.2
  3    ssd    1.74660   1.00000  1.7 TiB   1.7 GiB  1.7 GiB    1 KiB    34 MiB  1.7 TiB  0.10  0.04    1      up          osd.3
  4    ssd    1.74660   1.00000  1.7 TiB    32 GiB   32 GiB    4 KiB   334 MiB  1.7 TiB  1.79  0.74    3      up          osd.4
  5    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    3 KiB  1023 MiB  1.7 TiB  1.10  0.45    3      up          osd.5
  6    ssd    1.74660   1.00000  1.7 TiB    39 GiB   38 GiB    9 KiB   1.0 GiB  1.7 TiB  2.20  0.91    5      up          osd.6
  7    ssd    1.74660   1.00000  1.7 TiB    76 GiB   75 GiB   14 KiB   1.2 GiB  1.7 TiB  4.28  1.76    9      up          osd.7
  8    ssd    1.74660   1.00000  1.7 TiB    82 GiB   81 GiB   19 KiB  1004 MiB  1.7 TiB  4.56  1.88    7      up          osd.8
  9    ssd    1.86299   1.00000  1.9 TiB    36 GiB   35 GiB    7 KiB   898 MiB  1.8 TiB  1.88  0.78    4      up          osd.9
 10    ssd    1.74660   1.00000  1.7 TiB    62 GiB   62 GiB   16 KiB   449 MiB  1.7 TiB  3.48  1.44    5      up          osd.10
 11    ssd    1.74660   1.00000  1.7 TiB    70 GiB   68 GiB   12 KiB   1.4 GiB  1.7 TiB  3.90  1.61    7      up          osd.11
 12    ssd    1.74660   1.00000  1.7 TiB   5.7 GiB  5.2 GiB    2 KiB   545 MiB  1.7 TiB  0.32  0.13    2      up          osd.12
 13    ssd    1.74660   1.00000  1.7 TiB    64 GiB   63 GiB   15 KiB   985 MiB  1.7 TiB  3.57  1.47    5      up          osd.13
 14    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    8 KiB  1005 MiB  1.7 TiB  1.12  0.46    2      up          osd.14
 15    ssd    1.74660   1.00000  1.7 TiB    54 GiB   52 GiB   12 KiB   1.5 GiB  1.7 TiB  2.99  1.24    5      up          osd.15
 16    ssd    1.74660   1.00000  1.7 TiB   4.4 GiB  3.8 GiB    2 KiB   541 MiB  1.7 TiB  0.24  0.10    1      up          osd.16
 17    ssd    1.74660   1.00000  1.7 TiB    29 GiB   27 GiB    8 KiB   1.8 GiB  1.7 TiB  1.60  0.66    5      up          osd.17
 18    ssd    1.74660   1.00000  1.7 TiB    51 GiB   51 GiB    4 KiB   917 MiB  1.7 TiB  2.88  1.19    5      up          osd.18
 19    ssd    1.74660   1.00000  1.7 TiB    49 GiB   48 GiB    7 KiB   796 MiB  1.7 TiB  2.74  1.13    6      up          osd.19
 20    ssd    1.74660   1.00000  1.7 TiB    67 GiB   66 GiB   16 KiB   977 MiB  1.7 TiB  3.74  1.54    7      up          osd.20
 21    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    4 KiB   861 MiB  1.7 TiB  1.10  0.45    2      up          osd.21
 22    ssd    1.86299   1.00000  1.9 TiB    15 GiB   15 GiB    7 KiB   117 MiB  1.8 TiB  0.80  0.33    1      up          osd.22
 23    ssd    1.74660   1.00000  1.7 TiB    23 GiB   22 GiB    4 KiB   1.2 GiB  1.7 TiB  1.31  0.54    3      up          osd.23
 24    ssd    1.74660   1.00000  1.7 TiB    55 GiB   53 GiB    7 KiB   1.7 GiB  1.7 TiB  3.05  1.26    6      up          osd.24
 25    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    2 KiB   687 MiB  1.7 TiB  1.09  0.45    2      up          osd.25
 -5          43.66493         -   44 TiB   1.2 TiB  1.2 TiB  217 KiB    37 GiB   42 TiB  2.78  1.15    -              host DFX-VH-UT-US-VHOST0002
 26    ssd    1.74660   1.00000  1.7 TiB    39 GiB   37 GiB   11 KiB   1.7 GiB  1.7 TiB  2.17  0.89    5      up          osd.26
 27    ssd    1.74660   1.00000  1.7 TiB    24 GiB   23 GiB    6 KiB   737 MiB  1.7 TiB  1.34  0.55    4      up          osd.27
 28    ssd    1.74660   1.00000  1.7 TiB    71 GiB   70 GiB   14 KiB   1.2 GiB  1.7 TiB  3.99  1.65    8      up          osd.28
 29    ssd    1.74660   1.00000  1.7 TiB    70 GiB   68 GiB   11 KiB   1.8 GiB  1.7 TiB  3.90  1.61    7      up          osd.29
 30    ssd    1.74660   1.00000  1.7 TiB    17 GiB   16 GiB      0 B   510 MiB  1.7 TiB  0.94  0.39    2      up          osd.30
 31    ssd    1.74660   1.00000  1.7 TiB   6.4 GiB  5.1 GiB    2 KiB   1.3 GiB  1.7 TiB  0.36  0.15    2      up          osd.31
 32    ssd    1.74660   1.00000  1.7 TiB    54 GiB   53 GiB   18 KiB   1.5 GiB  1.7 TiB  3.02  1.25    5      up          osd.32
 33    ssd    1.74660   1.00000  1.7 TiB    77 GiB   76 GiB   10 KiB   1.1 GiB  1.7 TiB  4.31  1.78   10      up          osd.33
 34    ssd    1.74660   1.00000  1.7 TiB    81 GiB   79 GiB    9 KiB   1.9 GiB  1.7 TiB  4.55  1.88    8      up          osd.34
 35    ssd    1.74660   1.00000  1.7 TiB   9.0 GiB  7.3 GiB    6 KiB   1.7 GiB  1.7 TiB  0.50  0.21    3      up          osd.35
 36    ssd    1.74660   1.00000  1.7 TiB    32 GiB   30 GiB    7 KiB   1.8 GiB  1.7 TiB  1.77  0.73    3      up          osd.36
 37    ssd    1.74660   1.00000  1.7 TiB   109 GiB  108 GiB   19 KiB   1.8 GiB  1.6 TiB  6.12  2.52    9      up          osd.37
 38    ssd    1.74660   1.00000  1.7 TiB    55 GiB   53 GiB    9 KiB   1.7 GiB  1.7 TiB  3.05  1.26    6      up          osd.38
 39    ssd    1.74660   1.00000  1.7 TiB   6.2 GiB  5.1 GiB   10 KiB   1.1 GiB  1.7 TiB  0.34  0.14    2      up          osd.39
 40    ssd    1.74660   1.00000  1.7 TiB    83 GiB   81 GiB    9 KiB   1.8 GiB  1.7 TiB  4.64  1.91    7      up          osd.40
 41    ssd    1.74660   1.00000  1.7 TiB    50 GiB   49 GiB    5 KiB   895 MiB  1.7 TiB  2.82  1.16    6      up          osd.41
 42    ssd    1.74660   1.00000  1.7 TiB    31 GiB   29 GiB    8 KiB   1.8 GiB  1.7 TiB  1.74  0.72    3      up          osd.42
 43    ssd    1.74660   1.00000  1.7 TiB    33 GiB   31 GiB    1 KiB   1.7 GiB  1.7 TiB  1.82  0.75    4      up          osd.43
 44    ssd    1.74660   1.00000  1.7 TiB    62 GiB   61 GiB   26 KiB   1.4 GiB  1.7 TiB  3.47  1.43    5      up          osd.44
 45    ssd    1.74660   1.00000  1.7 TiB    32 GiB   31 GiB    3 KiB   833 MiB  1.7 TiB  1.81  0.74    3      up          osd.45
 46    ssd    1.74660   1.00000  1.7 TiB    83 GiB   82 GiB    8 KiB   1.6 GiB  1.7 TiB  4.65  1.92    7      up          osd.46
 47    ssd    1.74660   1.00000  1.7 TiB    23 GiB   22 GiB    1 KiB   1.5 GiB  1.7 TiB  1.29  0.53    4      up          osd.47
 48    ssd    1.74660   1.00000  1.7 TiB    63 GiB   61 GiB    6 KiB   1.8 GiB  1.7 TiB  3.53  1.46    4      up          osd.48
 49    ssd    1.74660   1.00000  1.7 TiB   113 GiB  111 GiB   16 KiB   1.8 GiB  1.6 TiB  6.33  2.61    9      up          osd.49
 50    ssd    1.74660   1.00000  1.7 TiB    21 GiB   19 GiB    2 KiB   1.6 GiB  1.7 TiB  1.15  0.48    3      up          osd.50
 -7          45.41153         -   45 TiB   1.2 TiB  1.2 TiB  200 KiB    37 GiB   44 TiB  2.62  1.08    -              host DFX-VH-UT-US-VHOST0003
 52    ssd    1.74660   1.00000  1.7 TiB    45 GiB   44 GiB    3 KiB   1.2 GiB  1.7 TiB  2.54  1.05    3      up          osd.52
 53    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    5 KiB   1.6 GiB  1.7 TiB  1.14  0.47    3      up          osd.53
 54    ssd    1.74660   1.00000  1.7 TiB    31 GiB   30 GiB    6 KiB   1.7 GiB  1.7 TiB  1.76  0.73    3      up          osd.54
 55    ssd    1.74660   1.00000  1.7 TiB    17 GiB   15 GiB      0 B   1.6 GiB  1.7 TiB  0.95  0.39    1      up          osd.55
 56    ssd    1.74660   1.00000  1.7 TiB    49 GiB   49 GiB    6 KiB   902 MiB  1.7 TiB  2.76  1.14    4      up          osd.56
 57    ssd    1.74660   1.00000  1.7 TiB    49 GiB   48 GiB    4 KiB   1.9 GiB  1.7 TiB  2.76  1.14    4      up          osd.57
 58    ssd    1.74660   1.00000  1.7 TiB   101 GiB   99 GiB    9 KiB   1.2 GiB  1.6 TiB  5.63  2.32    9      up          osd.58
 59    ssd    1.74660   1.00000  1.7 TiB    63 GiB   61 GiB    4 KiB   1.8 GiB  1.7 TiB  3.50  1.45    5      up          osd.59
 60    ssd    1.74660   1.00000  1.7 TiB    33 GiB   32 GiB    4 KiB   1.2 GiB  1.7 TiB  1.84  0.76    4      up          osd.60
 61    ssd    1.74660   1.00000  1.7 TiB    79 GiB   77 GiB    5 KiB   1.9 GiB  1.7 TiB  4.40  1.82    6      up          osd.61
 62    ssd    1.74660   1.00000  1.7 TiB    68 GiB   66 GiB   14 KiB   1.1 GiB  1.7 TiB  3.78  1.56    8      up          osd.62
 63    ssd    1.74660   1.00000  1.7 TiB    75 GiB   73 GiB    8 KiB   1.8 GiB  1.7 TiB  4.21  1.74    8      up          osd.63
 64    ssd    1.74660   1.00000  1.7 TiB    41 GiB   40 GiB   11 KiB   788 MiB  1.7 TiB  2.29  0.94    5      up          osd.64
 65    ssd    1.74660   1.00000  1.7 TiB    68 GiB   67 GiB   14 KiB   1.7 GiB  1.7 TiB  3.82  1.58    7      up          osd.65
 66    ssd    1.74660   1.00000  1.7 TiB    33 GiB   31 GiB    5 KiB   1.5 GiB  1.7 TiB  1.83  0.75    3      up          osd.66
 67    ssd    1.74660   1.00000  1.7 TiB    16 GiB   15 GiB    2 KiB   578 MiB  1.7 TiB  0.87  0.36    1      up          osd.67
 68    ssd    1.74660   1.00000  1.7 TiB    47 GiB   45 GiB   16 KiB   1.9 GiB  1.7 TiB  2.62  1.08    8      up          osd.68
 69    ssd    1.74660   1.00000  1.7 TiB    23 GiB   22 GiB    2 KiB   945 MiB  1.7 TiB  1.27  0.53    3      up          osd.69
 70    ssd    1.74660   1.00000  1.7 TiB    53 GiB   52 GiB    7 KiB   1.0 GiB  1.7 TiB  2.96  1.22    5      up          osd.70
 71    ssd    1.74660   1.00000  1.7 TiB    50 GiB   48 GiB   24 KiB   1.7 GiB  1.7 TiB  2.81  1.16    5      up          osd.71
 72    ssd    1.74660   1.00000  1.7 TiB   7.8 GiB  6.5 GiB    2 KiB   1.3 GiB  1.7 TiB  0.44  0.18    3      up          osd.72
 73    ssd    1.74660   1.00000  1.7 TiB    45 GiB   44 GiB   17 KiB   1.8 GiB  1.7 TiB  2.54  1.05    6      up          osd.73
 74    ssd    1.74660   1.00000  1.7 TiB    21 GiB   19 GiB    1 KiB   1.7 GiB  1.7 TiB  1.16  0.48    4      up          osd.74
 75    ssd    1.74660   1.00000  1.7 TiB    40 GiB   38 GiB    5 KiB   1.7 GiB  1.7 TiB  2.23  0.92    6      up          osd.75
 76    ssd    1.74660   1.00000  1.7 TiB    32 GiB   30 GiB    4 KiB   1.7 GiB  1.7 TiB  1.77  0.73    2      up          osd.76
 77    ssd    1.74660   1.00000  1.7 TiB   112 GiB  111 GiB   22 KiB   1.2 GiB  1.6 TiB  6.27  2.59    9      up          osd.77

Find attached the PG distribution of my pools


Regards,

JP
 

Attachments

  • pools_pgs.png
    pools_pgs.png
    16.8 KB · Views: 9
Here is the output from the remaining 3 hosts:

Code: 
-9          45.99350         -   46 TiB   1.2 TiB  1.1 TiB  200 KiB    38 GiB   45 TiB  2.53  1.04    -              host DFX-VH-UT-US-VHOST0004

 78    ssd    1.86299   1.00000  1.9 TiB    53 GiB   52 GiB    5 KiB   1.8 GiB  1.8 TiB  2.80  1.16    6      up          osd.78

 79    ssd    1.86299   1.00000  1.9 TiB   107 GiB  105 GiB   11 KiB   2.0 GiB  1.8 TiB  5.58  2.30   11      up          osd.79

 80    ssd    1.74660   1.00000  1.7 TiB    70 GiB   68 GiB   13 KiB   1.4 GiB  1.7 TiB  3.90  1.61    8      up          osd.80

 81    ssd    1.74660   1.00000  1.7 TiB    35 GiB   33 GiB   10 KiB   1.8 GiB  1.7 TiB  1.97  0.81    3      up          osd.81

 82    ssd    1.74660   1.00000  1.7 TiB    76 GiB   74 GiB    4 KiB   1.8 GiB  1.7 TiB  4.26  1.76    5      up          osd.82

 83    ssd    1.74660   1.00000  1.7 TiB    65 GiB   63 GiB   17 KiB   1.6 GiB  1.7 TiB  3.62  1.50    5      up          osd.83

 84    ssd    1.74660   1.00000  1.7 TiB    16 GiB   15 GiB    4 KiB  1006 MiB  1.7 TiB  0.89  0.37    2      up          osd.84

 85    ssd    1.74660   1.00000  1.7 TiB    62 GiB   61 GiB    4 KiB   879 MiB  1.7 TiB  3.46  1.43    5      up          osd.85

 86    ssd    1.74660   1.00000  1.7 TiB    32 GiB   31 GiB    1 KiB   813 MiB  1.7 TiB  1.76  0.73    3      up          osd.86

 87    ssd    1.74660   1.00000  1.7 TiB    58 GiB   57 GiB   14 KiB   1.9 GiB  1.7 TiB  3.27  1.35    8      up          osd.87

 88    ssd    1.74660   1.00000  1.7 TiB    72 GiB   70 GiB   10 KiB   1.3 GiB  1.7 TiB  4.00  1.65    8      up          osd.88

 89    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    4 KiB   1.5 GiB  1.7 TiB  1.12  0.46    2      up          osd.89

 90    ssd    1.74660   1.00000  1.7 TiB    57 GiB   55 GiB    4 KiB   1.9 GiB  1.7 TiB  3.18  1.31    7      up          osd.90

 91    ssd    1.74660   1.00000  1.7 TiB    20 GiB   18 GiB    2 KiB   1.3 GiB  1.7 TiB  1.10  0.46    3      up          osd.91

 92    ssd    1.74660   1.00000  1.7 TiB    54 GiB   52 GiB    8 KiB   1.5 GiB  1.7 TiB  3.00  1.24    5      up          osd.92

 93    ssd    1.74660   1.00000  1.7 TiB    48 GiB   46 GiB    3 KiB   1.6 GiB  1.7 TiB  2.68  1.10    5      up          osd.93

 94    ssd    1.74660   1.00000  1.7 TiB    35 GiB   33 GiB    5 KiB   1.5 GiB  1.7 TiB  1.93  0.80    3      up          osd.94

 95    ssd    1.74660   1.00000  1.7 TiB   1.9 GiB  1.6 GiB      0 B   303 MiB  1.7 TiB  0.11  0.04    1      up          osd.95

 96    ssd    1.74660   1.00000  1.7 TiB    18 GiB   17 GiB    5 KiB   1.6 GiB  1.7 TiB  1.02  0.42    2      up          osd.96

 97    ssd    1.86299   1.00000  1.9 TiB    48 GiB   47 GiB   24 KiB   1.2 GiB  1.8 TiB  2.53  1.04    6      up          osd.97

 98    ssd    1.86299   1.00000  1.9 TiB    54 GiB   52 GiB    7 KiB   1.6 GiB  1.8 TiB  2.81  1.16    5      up          osd.98

 99    ssd    1.86299   1.00000  1.9 TiB    57 GiB   56 GiB   18 KiB   1.7 GiB  1.8 TiB  3.01  1.24    6      up          osd.99

100    ssd    1.74660   1.00000  1.7 TiB    37 GiB   35 GiB    5 KiB   1.7 GiB  1.7 TiB  2.06  0.85    5      up          osd.100

101    ssd    1.74660   1.00000  1.7 TiB    50 GiB   49 GiB    7 KiB   1.7 GiB  1.7 TiB  2.82  1.16    4      up          osd.101

102    ssd    1.74660   1.00000  1.7 TiB   8.4 GiB  7.2 GiB    4 KiB   1.1 GiB  1.7 TiB  0.47  0.19    2      up          osd.102

103    ssd    1.74660   1.00000  1.7 TiB    39 GiB   38 GiB   11 KiB   1.7 GiB  1.7 TiB  2.19  0.91    7      up          osd.103

-11          45.41153         -   45 TiB   1.1 TiB  1.0 TiB  235 KiB    37 GiB   44 TiB  2.33  0.96    -              host DFX-VH-UT-US-VHOST0005

104    ssd    1.74660   1.00000  1.7 TiB    34 GiB   34 GiB    7 KiB   780 MiB  1.7 TiB  1.92  0.79    3      up          osd.104

105    ssd    1.74660   1.00000  1.7 TiB    50 GiB   48 GiB    5 KiB   1.7 GiB  1.7 TiB  2.79  1.15    4      up          osd.105

106    ssd    1.74660   1.00000  1.7 TiB    69 GiB   69 GiB   12 KiB   905 MiB  1.7 TiB  3.88  1.60    7      up          osd.106

107    ssd    1.74660   1.00000  1.7 TiB   6.4 GiB  5.1 GiB    3 KiB   1.3 GiB  1.7 TiB  0.36  0.15    2      up          osd.107

108    ssd    1.74660   1.00000  1.7 TiB    23 GiB   22 GiB    5 KiB  1023 MiB  1.7 TiB  1.28  0.53    5      up          osd.108

109    ssd    1.74660   1.00000  1.7 TiB   104 GiB  103 GiB   20 KiB   1.3 GiB  1.6 TiB  5.81  2.40   11      up          osd.109

110    ssd    1.74660   1.00000  1.7 TiB    35 GiB   33 GiB    4 KiB   1.7 GiB  1.7 TiB  1.95  0.81    4      up          osd.110

111    ssd    1.74660   1.00000  1.7 TiB    25 GiB   23 GiB    5 KiB   1.2 GiB  1.7 TiB  1.37  0.57    4      up          osd.111

112    ssd    1.74660   1.00000  1.7 TiB    56 GiB   55 GiB   12 KiB   907 MiB  1.7 TiB  3.13  1.29    6      up          osd.112

113    ssd    1.74660   1.00000  1.7 TiB    47 GiB   45 GiB    8 KiB   1.7 GiB  1.7 TiB  2.61  1.08    4      up          osd.113

114    ssd    1.74660   1.00000  1.7 TiB    51 GiB   51 GiB   16 KiB   844 MiB  1.7 TiB  2.87  1.19    5      up          osd.114

115    ssd    1.74660   1.00000  1.7 TiB    53 GiB   52 GiB   12 KiB   1.7 GiB  1.7 TiB  2.98  1.23    5      up          osd.115

116    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB   12 KiB   1.6 GiB  1.7 TiB  1.14  0.47    3      up          osd.116

117    ssd    1.74660   1.00000  1.7 TiB    21 GiB   19 GiB    7 KiB   1.8 GiB  1.7 TiB  1.16  0.48    2      up          osd.117

118    ssd    1.74660   1.00000  1.7 TiB    20 GiB   18 GiB    3 KiB   1.4 GiB  1.7 TiB  1.10  0.46    2      up          osd.118

119    ssd    1.74660   1.00000  1.7 TiB    24 GiB   22 GiB    9 KiB   1.8 GiB  1.7 TiB  1.34  0.55    4      up          osd.119

120    ssd    1.74660   1.00000  1.7 TiB    46 GiB   45 GiB   13 KiB   1.5 GiB  1.7 TiB  2.59  1.07    3      up          osd.120

121    ssd    1.74660   1.00000  1.7 TiB    39 GiB   38 GiB   21 KiB   1.7 GiB  1.7 TiB  2.20  0.91    7      up          osd.121

122    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB   11 KiB   1.0 GiB  1.7 TiB  1.11  0.46    2      up          osd.122

123    ssd    1.74660   1.00000  1.7 TiB    31 GiB   30 GiB    5 KiB  1007 MiB  1.7 TiB  1.76  0.72    2      up          osd.123

124    ssd    1.74660   1.00000  1.7 TiB    46 GiB   45 GiB    4 KiB   1.8 GiB  1.7 TiB  2.59  1.07    3      up          osd.124

125    ssd    1.74660   1.00000  1.7 TiB    38 GiB   36 GiB    6 KiB   1.9 GiB  1.7 TiB  2.11  0.87    5      up          osd.125

126    ssd    1.74660   1.00000  1.7 TiB   110 GiB  108 GiB    8 KiB   1.3 GiB  1.6 TiB  6.14  2.53    9      up          osd.126

127    ssd    1.74660   1.00000  1.7 TiB   5.3 GiB  3.7 GiB    3 KiB   1.5 GiB  1.7 TiB  0.29  0.12    1      up          osd.127

128    ssd    1.74660   1.00000  1.7 TiB    35 GiB   34 GiB    7 KiB   1.8 GiB  1.7 TiB  1.98  0.82    4      up          osd.128

129    ssd    1.74660   1.00000  1.7 TiB    74 GiB   73 GiB   17 KiB   1.3 GiB  1.7 TiB  4.16  1.72    9      up          osd.129

-13          47.04105         -   47 TiB  1020 GiB  983 GiB  238 KiB    36 GiB   46 TiB  2.12  0.87    -              host DFX-VH-UT-US-VHOST0006

130    ssd    1.74660   1.00000  1.7 TiB    15 GiB   15 GiB    3 KiB   602 MiB  1.7 TiB  0.85  0.35    1      up          osd.130

131    ssd    1.74660   1.00000  1.7 TiB   9.4 GiB  8.6 GiB    6 KiB   834 MiB  1.7 TiB  0.52  0.22    4      up          osd.131

132    ssd    1.86299   1.00000  1.9 TiB    40 GiB   39 GiB    8 KiB   1.7 GiB  1.8 TiB  2.11  0.87    5      up          osd.132

133    ssd    1.86299   1.00000  1.9 TiB    35 GiB   33 GiB    9 KiB   1.5 GiB  1.8 TiB  1.83  0.76    3      up          osd.133

134    ssd    1.86299   1.00000  1.9 TiB    17 GiB   16 GiB    9 KiB   1.4 GiB  1.8 TiB  0.91  0.37    1      up          osd.134

135    ssd    1.86299   1.00000  1.9 TiB   103 GiB  102 GiB   13 KiB   1.7 GiB  1.8 TiB  5.43  2.24   11      up          osd.135

136    ssd    1.86299   1.00000  1.9 TiB   8.1 GiB  6.7 GiB    4 KiB   1.5 GiB  1.9 TiB  0.43  0.18    4      up          osd.136

137    ssd    1.86299   1.00000  1.9 TiB    54 GiB   52 GiB   11 KiB   1.6 GiB  1.8 TiB  2.81  1.16    5      up          osd.137

138    ssd    1.86299   1.00000  1.9 TiB    39 GiB   38 GiB    8 KiB   823 MiB  1.8 TiB  2.02  0.83    6      up          osd.138

139    ssd    1.74660   1.00000  1.7 TiB    57 GiB   56 GiB    9 KiB   1.1 GiB  1.7 TiB  3.19  1.32    6      up          osd.139

140    ssd    1.74660   1.00000  1.7 TiB    33 GiB   32 GiB   10 KiB   1.6 GiB  1.7 TiB  1.86  0.77    5      up          osd.140

141    ssd    1.86299   1.00000  1.9 TiB    44 GiB   42 GiB   18 KiB   1.7 GiB  1.8 TiB  2.30  0.95    7      up          osd.141

142    ssd    1.86299   1.00000  1.9 TiB    18 GiB   17 GiB    4 KiB   1.6 GiB  1.8 TiB  0.96  0.39    2      up          osd.142

143    ssd    1.86299   1.00000  1.9 TiB   9.5 GiB  8.6 GiB    6 KiB   995 MiB  1.9 TiB  0.50  0.21    5      up          osd.143

144    ssd    1.86299   1.00000  1.9 TiB    38 GiB   37 GiB   12 KiB   960 MiB  1.8 TiB  1.98  0.82    4      up          osd.144

145    ssd    1.74660   1.00000  1.7 TiB    61 GiB   59 GiB   13 KiB   1.7 GiB  1.7 TiB  3.42  1.41    4      up          osd.145

146    ssd    1.74660   1.00000  1.7 TiB    54 GiB   53 GiB    8 KiB   1.1 GiB  1.7 TiB  3.02  1.25    5      up          osd.146

147    ssd    1.74660   1.00000  1.7 TiB    52 GiB   50 GiB   17 KiB   1.8 GiB  1.7 TiB  2.89  1.19    5      up          osd.147

148    ssd    1.74660   1.00000  1.7 TiB    58 GiB   57 GiB   13 KiB   1.8 GiB  1.7 TiB  3.27  1.35    7      up          osd.148

149    ssd    1.74660   1.00000  1.7 TiB    32 GiB   30 GiB    5 KiB   1.3 GiB  1.7 TiB  1.77  0.73    2      up          osd.149

150    ssd    1.74660   1.00000  1.7 TiB    20 GiB   19 GiB    7 KiB   1.0 GiB  1.7 TiB  1.12  0.46    2      up          osd.150

151    ssd    1.74660   1.00000  1.7 TiB   5.4 GiB  3.8 GiB    8 KiB   1.6 GiB  1.7 TiB  0.30  0.13    2      up          osd.151

152    ssd    1.74660   1.00000  1.7 TiB    33 GiB   32 GiB    6 KiB   1.4 GiB  1.7 TiB  1.85  0.76    3      up          osd.152

153    ssd    1.86299   1.00000  1.9 TiB    22 GiB   20 GiB    6 KiB   1.6 GiB  1.8 TiB  1.13  0.47    3      up          osd.153

154    ssd    1.86299   1.00000  1.9 TiB    64 GiB   62 GiB   13 KiB   1.8 GiB  1.8 TiB  3.37  1.39    5      up          osd.154

155    ssd    1.86299   1.00000  1.9 TiB    98 GiB   96 GiB   12 KiB   1.8 GiB  1.8 TiB  5.14  2.12    9      up          osd.155

                          TOTAL  273 TiB   6.6 TiB  6.4 TiB  1.3 MiB   209 GiB  267 TiB  2.42

MIN/MAX VAR: 0.04/2.61  STDDEV: 1.42
 
Sorry, I've been on vacation and have just replied to you.

Your PGS distribution is not evenly distributed, and the number of PGS on each osd is a bit too small.
If it were me, I would try to increase the number of PGS in the pool (this requires good calculations) and try to evenly distribute the number of PGS on each OSD. When doing the above operations, it will affect the read and write speed and efficiency of the cluster, so it should be done when no one is using it. And be careful.
 
rados bench -p testbench 10 rand
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 15 345 330 1319.3 1320 0.0220504 0.0458886
2 16 682 666 1331.51 1344 0.0484838 0.0460847
3 15 1022 1007 1342.18 1364 0.0657176 0.0463005
4 16 1363 1347 1346.58 1360 0.0272982 0.0463732
5 16 1711 1695 1355.61 1392 0.024731 0.0458902
6 16 2060 2044 1362.3 1396 0.0249081 0.0459957
7 16 2404 2388 1364.22 1376 0.0217204 0.0458798
8 16 2740 2724 1361.66 1344 0.0646662 0.0459278
9 16 3093 3077 1367.22 1412 0.026175 0.0458007
10 14 3445 3431 1372.07 1416 0.0401794 0.0456587
Total time run: 10.0358
Total reads made: 3445
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1373.08
Average IOPS: 343
Stddev IOPS: 7.9085
Max IOPS: 354
Min IOPS: 330
Average Latency(s): 0.045714
Max latency(s): 0.353993
Min latency(s): 0.00458857

The read benchmark indicates that the bandwidth is being throttled. 10Gbit across 6 hosts is insufficient; I believe you need at least 25Gbit. As Wuwzy mentioned, you need at least 100 PGs per OSD to obtain reliable results. The variation will automatically improve, and the benchmark values for writing should slightly increase. Since you've already reached the network’s maximum for reading, not much more will change in that area.
 
wuwzy said:
Sorry, I've been on vacation and have just replied to you.

Your PGS distribution is not evenly distributed, and the number of PGS on each osd is a bit too small.
If it were me, I would try to increase the number of PGS in the pool (this requires good calculations) and try to evenly distribute the number of PGS on each OSD. When doing the above operations, it will affect the read and write speed and efficiency of the cluster, so it should be done when no one is using it. And be careful.
Click to expand...
Hello @wuwzy

No worries, thanks for taking the time to look into this,

Based on https://docs.ceph.com/en/squid/rados/operations/pgcalc/ I should set the PG count to 4096

Cluster is not in production so it should not be an issue to change this right now

Thanks for the recommendation, it does make sense, hope it improves this
 

Attachments

  • rbd_pool_pgs.png
    rbd_pool_pgs.png
    9.5 KB · Views: 5
Last edited:
  • Like
Reactions: wuwzy
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom