Slow write on a VM

HPmoss · Mar 22, 2024

I've noticed that I am getting choppy and slow file transfer speed (from local computer to a Proxmox hosted VM). On other hand, pulling large files from there go on full speed but sending data there is slow. I am wondering if the SSD/HDD is being a bottleneck. Performance is same for Linux of Windows based VM so it's not the OS that's causing slowness. How do I go about troubleshooting this one?

Version:
pve-manager/8.1.5/60e01c6ac2325b3f (running kernel: 6.5.13-3-pve)

Dunuin · Mar 22, 2024

Use "fio" for benchmarking the storage and "iperf" for benchmarking the network.

Keep in mind that SMR HDDs and QLC SSDs are terrible at writing and a big performance drop is totally expected.

Kingneutron · Mar 22, 2024

Use virtio drivers in the VM and turn off all the hardware-assisted (offload) stuff for the in-vm NIC. Also set multi-queue at the proxmox GUI level to match the number of virtual CPUs you gave the VM

HPmoss · Mar 23, 2024

Dunuin said:
Use "fio" for benchmarking the storage and "iperf" for benchmarking the network.

Keep in mind that SMR HDDs and QLC SSDs are terrible at writing and a big performance drop is totally expected.

iperf came out OK. However I am not familiar with "fio". I will get a hold on to it.
But do tell me when baseline or numbers am I looking for to constitute normality?

Dunuin · Mar 23, 2024

HPmoss said:
But do tell me when baseline or numbers am I looking for to constitute normality?

Something like 40MB/s when doing big sequential async writes or to an QLC SSD or 1-2 MB/s when doing 4K sync writes to any consumer SSD would be totally normal. HDDs could be as slow as 400kb/s.

When buying new disks you basically get what you pay for. If the price per GB is great, the write performance won't. There is a reason why you could buy a same sized SSD for 30$ or 600$. Like an 4-core N100 can't compete with a 64-core Xeon. But people way too often still buy the cheapest SSDs they could find and dump all the saved money in bigger GPUs or CPUs or RGB

HPmoss · Mar 23, 2024

Disk in question:
WD 3TB

https://hdd.userbenchmark.com/WD-Blue-3TB-2015/Rating/3522

Result:
$ fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio, iodepth=1
fio-3.35
Starting 1 process
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [F(1)][100.0%][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=36790: Sat Mar 23 13:44:02 2024
write: IOPS=4441, BW=17.3MiB/s (18.2MB/s)(2677MiB/154310msec); 0 zone resets
slat (nsec): min=777, max=3291.6k, avg=3339.05, stdev=5534.31
clat (nsec): min=326, max=26603k, avg=83280.84, stdev=826982.06
lat (usec): min=7, max=26605, avg=86.62, stdev=827.02
clat percentiles (usec):
| 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18],
| 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 20],
| 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 58],
| 99.00th=[ 143], 99.50th=[ 6718], 99.90th=[12911], 99.95th=[16450],
| 99.99th=[22676]
bw ( KiB/s): min=10864, max=179736, per=100.00%, avg=45897.41, stdev=56153.97, samples=119
iops : min= 2716, max=44934, avg=11474.35, stdev=14038.49, samples=119
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.13%, 20=69.59%, 50=24.31%
lat (usec) : 100=4.04%, 250=1.27%, 500=0.04%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.41%, 20=0.16%, 50=0.03%
cpu : usr=1.41%, sys=1.92%, ctx=689495, majf=0, minf=16
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,685373,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=17.3MiB/s (18.2MB/s), 17.3MiB/s-17.3MiB/s (18.2MB/s-18.2MB/s), io=2677MiB (2807MB), run=154310-154310msec

Disk stats (read/write):
dm-0: ios=2/247832, merge=0/0, ticks=86/2637960, in_queue=2638046, util=97.39%, aggrios=16/246355, aggrmerge=0/1512, aggrticks=698/2339625, aggrin_queue=2346692, aggrutil=97.48%
sda: ios=16/246355, merge=0/1512, ticks=698/2339625, in_queue=2346692, util=97.48%

Dunuin · Mar 23, 2024

Thats way too fast for that HDD. You probably got PVE installed on some consumer SSD and you are actually benchmarking that and not the HDD? As you didn't specify any target disk/filesystem to benchmark.

HPmoss · Mar 25, 2024

Dunuin said:
Thats way too fast for that HDD. You probably got PVE installed on some consumer SSD and you are actually benchmarking that and not the HDD? As you didn't specify any target disk/filesystem to benchmark.

I am using it as per documentation, a directory is created on the partition on that 3TB HDD (the disk I wanted to test). In this case it is /home/Media/fioTEST

Do let me know if there's better way to run fio.

Results are attached.

HPmoss · Mar 25, 2024

In case this helps ...

spirit · Mar 25, 2024

"--ioengine libaio --direct 1 --sync 1"

HPmoss · Mar 25, 2024

Here is the result:

Code:

$ pwd
/home/Media/fioTEST
$ fio --name=random-write --ioengine=libaio --direct 1 --sync 1 --rw=randwrite --bs=4k --numjobs=1 --size=8g --iodepth=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=128KiB/s][w=32 IOPS][eta 00m:00s]
random-write: (groupid=0, jobs=1): err= 0: pid=155466: Mon Mar 25 11:22:56 2024
  write: IOPS=25, BW=101KiB/s (103kB/s)(6032KiB/60005msec); 0 zone resets
    slat (usec): min=17, max=121, avg=59.68, stdev=10.15
    clat (msec): min=7, max=309, avg=39.72, stdev=19.58
     lat (msec): min=7, max=309, avg=39.78, stdev=19.58
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   12], 20.00th=[   34],
     | 30.00th=[   45], 40.00th=[   45], 50.00th=[   45], 60.00th=[   45],
     | 70.00th=[   45], 80.00th=[   45], 90.00th=[   45], 95.00th=[   45],
     | 99.00th=[   67], 99.50th=[   89], 99.90th=[  305], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=   56, max=  328, per=98.48%, avg=99.83, stdev=41.65, samples=119
   iops        : min=   14, max=   82, avg=24.96, stdev=10.41, samples=119
  lat (msec)   : 10=6.56%, 20=8.62%, 50=82.56%, 100=1.86%, 250=0.07%
  lat (msec)   : 500=0.33%
  cpu          : usr=0.07%, sys=0.16%, ctx=1512, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1508,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=101KiB/s (103kB/s), 101KiB/s-101KiB/s (103kB/s-103kB/s), io=6032KiB (6177kB), run=60005-60005msec

Disk stats (read/write):
    dm-2: ios=0/4508, merge=0/0, ticks=0/59440, in_queue=59440, util=99.97%, aggrios=0/4547, aggrmerge=0/0, aggrticks=0/60171, aggrin_queue=118480, aggrutil=99.80%
  sdb: ios=0/4547, merge=0/0, ticks=0/60171, in_queue=118480, util=99.80%

spirit · Mar 25, 2024

w=128KiB/s][w=32 IOPS ....

maybe try to increase iodepth a little bit, but if it's an hdd, you shouldn't be able to reach more than 150~200 iops

HPmoss · Mar 25, 2024

On this HDD I am getting following with iodepth at 190

Run status group 0 (all jobs):
WRITE: bw=320KiB/s (328kB/s), 320KiB/s-320KiB/s (328kB/s-328kB/s), io=20.1MiB (21.1MB), run=64341-64341msec

spirit · Mar 25, 2024

maybe 190 is a little bit high, what is the result with 32 or 64 queues ?

you should really look if you don't have a SMR disk

for example:
https://www.paradigit.be/western-digital-blue-smr-2-tb/20975485/product

because you really don't want to use this for vms. (they are only good for cold data)

HPmoss · Mar 25, 2024

At --iodepth=64 following is the end result:

WRITE: bw=219KiB/s (224kB/s), 219KiB/s-219KiB/s (224kB/s-224kB/s), io=12.9MiB (13.5MB), run=60438-60438msec

It is probably a CMR disk as per documentation. This Proxmox system is not something that I have designed from ground up. It's a test-lab kind of system based on a old inherited HP z230 workstation and a bunch of disks that were lying around at the time. It serves the purpose well for a small SoHo with file storage and media server (Emby/Plex) and few VMs.

spirit · Mar 25, 2024

it's a 5400rpm ,so it's still slow than a 7200rpm.

224kB/s at 4k , it's around 56 iops . (I think they could reach 100~120 iops at 5400rpm)

for fileserver it can be ok, but for vm (if you have multiple vms, they are send each their sync at different times. Do the more vm you'll have in parralel, the more fsync you'll have).

maybe try with --sync option, as if you don't have too many vm, you'll not have a sync at each write.

Dunuin · Mar 25, 2024

And fio was done on top of the filesystem. So you won't reach those 100-120 IOPS because of the metadata and whatever adds overhead and amplifies writes.

HPmoss · Mar 25, 2024

There's no VM on this HDD. Entire disk is attached to a VM for file storage purpose. I am using AlmaLinux 9.3 stored on a separate SSD.

Slow write on a VM

Member

Distinguished Member

Active Member

Member

Distinguished Member

Member

Distinguished Member

Member

Attachments

Member

Attachments

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Attachments

Distinguished Member

Distinguished Member

Member