HDD Disk Performance Terribly Slow on Guest OS, within 100KiB/sec

H

HOchacha

New Member
Mar 9, 2025
2
0
1
Hi, I recently move to proxmox from VMware.
My System Spec is enough to use.

Bash: 
NAME                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                    8:0    0   9.1T  0 disk
└─sda2                 8:2    0   9.1T  0 part /mnt/data
nvme0n1              259:0    0 465.8G  0 disk
├─nvme0n1p1          259:1    0  1007K  0 part
├─nvme0n1p2          259:2    0     1G  0 part /boot/efi
└─nvme0n1p3          259:3    0 464.8G  0 part
  ├─pve-swap         252:0    0     8G  0 lvm 
  ├─pve-root         252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta   252:2    0   3.4G  0 lvm 
  │ └─pve-data-tpool 252:4    0 337.9G  0 lvm 
  │   └─pve-data     252:5    0 337.9G  1 lvm 
  └─pve-data_tdata   252:3    0 337.9G  0 lvm 
    └─pve-data-tpool 252:4    0 337.9G  0 lvm 
      └─pve-data     252:5    0 337.9G  1 lvm

However, the performance on a Guest VM is terribly slow to use because of the Disk Performance.
  • FIO read (cached)
    • BW: avg 182 MiB/sec, min 37MiB/sec, max 251MiB/sec, stdev 43MiB/sec
    • IOPS: avg 46139, min 9472, max 62788, stdev 10867.85
  • FIO randread (cached)
    • BW: avg 511.66KiB/sec, min 88KiB/sec, max 648KiB/sec, stdev 111.54KiB/sec
    • IOPS: avg 134.92, min 82, max 154, avg 134.92, stdev=20.09,

Can i have any clues for resolve this problem???
 
Thank you for replying my thread.

First of all, Spec

Physical HW
Intel Xeon Gold 5220R 2.20Hz 2 Socket
Memory 187GiB
SSD Samsung SSD 980 465.8GiB
HDD TOSHIBA MG06ACA10TE 9.1TiB

VM
Virtual CPU 16 core 1socket
Mem 16GiB
HDD 300GiB
- SCSI Controller: VirtIO SCSI
- Hard Disk: iothread=1

Fio Command (for read/write)
Bash: 
boan@VSC-32224444:~$ fio --name=test --rw=write --direct=1 --bs=16k --direct=1 --size=1G --iodepth=4
test: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1440KiB/s][w=90 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=630110: Mon Mar 10 04:25:31 2025
  write: IOPS=97, BW=1556KiB/s (1593kB/s)(1024MiB/673936msec); 0 zone resets
    clat (usec): min=162, max=355169, avg=10280.02, stdev=21847.76
     lat (usec): min=162, max=355169, avg=10280.63, stdev=21847.83
    clat percentiles (usec):
     |  1.00th=[   235],  5.00th=[   334], 10.00th=[   416], 20.00th=[   562],
     | 30.00th=[   685], 40.00th=[   791], 50.00th=[   914], 60.00th=[  1074],
     | 70.00th=[  1434], 80.00th=[  2507], 90.00th=[ 45351], 95.00th=[ 52167],
     | 99.00th=[ 80217], 99.50th=[123208], 99.90th=[177210], 99.95th=[193987],
     | 99.99th=[231736]
   bw (  KiB/s): min=  192, max=18240, per=100.00%, avg=1556.83, stdev=1686.08, samples=1347
   iops        : min=   12, max= 1140, avg=97.23, stdev=105.30, samples=1347
  lat (usec)   : 250=1.53%, 500=13.92%, 750=20.69%, 1000=20.09%
  lat (msec)   : 2=21.81%, 4=3.30%, 10=0.22%, 20=0.04%, 50=12.33%
  lat (msec)   : 100=5.35%, 250=0.70%, 500=0.01%
  cpu          : usr=0.08%, sys=0.35%, ctx=65606, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=1556KiB/s (1593kB/s), 1556KiB/s-1556KiB/s (1593kB/s-1593kB/s), io=1024MiB (1074MB), run=673936-673936msec

Disk stats (read/write):
    dm-0: ios=0/66779, merge=0/0, ticks=0/718804, in_queue=718804, util=99.84%, aggrios=0/66328, aggrmerge=0/477, aggrticks=0/706961, aggrin_queue=722570, aggrutil=99.86%
  sda: ios=0/66328, merge=0/477, ticks=0/706961, in_queue=722570, util=99.86%
Bash: 
boan@VSC-32224444:~$ fio --name=test --rw=read --direct=1 --bs=16k --direct=1 --size=1G --iodepth=4
test: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=4
fio-3.28
Starting 1 process
Jobs: 1 (f=1): [R(1)][99.9%][r=2672KiB/s][r=167 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=630259: Mon Mar 10 04:50:43 2025
  read: IOPS=65, BW=1046KiB/s (1072kB/s)(1024MiB/1002019msec)
    clat (usec): min=140, max=275143, avg=15286.70, stdev=28752.39
     lat (usec): min=140, max=275143, avg=15286.98, stdev=28752.50
    clat percentiles (usec):
     |  1.00th=[   198],  5.00th=[   241], 10.00th=[   302], 20.00th=[   371],
     | 30.00th=[   400], 40.00th=[   433], 50.00th=[   482], 60.00th=[   685],
     | 70.00th=[  1004], 80.00th=[ 50070], 90.00th=[ 58459], 95.00th=[ 66847],
     | 99.00th=[108528], 99.50th=[175113], 99.90th=[217056], 99.95th=[233833],
     | 99.99th=[258999]
   bw (  KiB/s): min=  128, max=25408, per=99.76%, avg=1044.72, stdev=2058.25, samples=2003
   iops        : min=    8, max= 1588, avg=65.26, stdev=128.57, samples=2003
  lat (usec)   : 250=5.93%, 500=46.28%, 750=10.41%, 1000=7.30%
  lat (msec)   : 2=4.05%, 4=0.77%, 10=0.40%, 20=0.18%, 50=5.53%
  lat (msec)   : 100=18.05%, 250=1.07%, 500=0.02%
  cpu          : usr=0.05%, sys=0.21%, ctx=65542, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=65536,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=1046KiB/s (1072kB/s), 1046KiB/s-1046KiB/s (1072kB/s-1072kB/s), io=1024MiB (1074MB), run=1002019-1002019msec

Disk stats (read/write):
    dm-0: ios=65436/1583, merge=0/0, ticks=999668/71476, in_queue=1071144, util=99.20%, aggrios=65536/968, aggrmerge=0/615, aggrticks=1000226/49069, aggrin_queue=1065330, aggrutil=99.21%
  sda: ios=65536/968, merge=0/615, ticks=1000226/49069, in_queue=1065330, util=99.21%
 
You must log in or register to reply here.
Top Bottom