Extremely Slow SSD Write Speed in VM

D

daNutz

New Member
Mar 24, 2023
18
0
1
Hi,

Im experiencing very slow disk performance in a VM, the disk is connected to this VM only.

PXE Host:
root@proxmox-02:/dev/HDD1TB# dd if=/dev/zero of=/dev/SSD2TB/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.16671 s, 920 MB/s

VM (Ubuntu):
docker@docker-02:~$ sudo dd if=/dev/zero of=/mnt/SSD2TB/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.6508 s, 52.0 MB/s

VM Disk Config: https://imgur.com/a/KCS60WN

No bandwidth limitations set.
 
daNutz said:
thanks but irrelevant right now as its performing very well in the host.
Click to expand...
Until the write buffer get full and it drops to KB/s.
EDIT: I'm happy for you that you don't have the problems that all other people here have with QLC drives. It's just hard to believe for me.
 
Last edited:
daNutz said:
thanks but irrelevant right now as its performing very well in the host.
Click to expand...
Yes, unrealistically high ... so high that it is not physically possible with SATA 3. And writing zeros is not test. Please use fio or just believe what people with a lot of experience (@leesteken and I) say ... or search the forum and join the other non-believers that have been disproven.
 
  • Like
Reactions: t.lamprecht and itNGO
LnxBil said:
Yes, unrealistically high ... so high that it is not physically possible with SATA 3. And writing zeros is not test. Please use fio or just believe what people with a lot of experience (@leesteken and I) say ... or search the forum and join the other non-believers that have been disproven.
Click to expand...
im not saying i dont believe, however im not seeing any data relating to that being the cause of my current issue.
 
daNutz said:
im not saying i dont believe, however im not seeing any data relating to that being the cause of my current issue.
Click to expand...
It looks like you are running non-realistic tests and getting strange results that cannot be fully explained. Running fio tests will give more realistic and comparable results, but those will probably make you very unhappy. Please have a look at other threads and the fio tests that were run on QLC drives. It's always sad to inform people that they wasted their money on QLC drives when doing (enterprise) virtualization and often people don't accept it. Maybe we can turn things around and you can simply prove that this problem does not affect you by convincing people with similar fio tests?
 
  • Like
Reactions: daNutz
Not sure if ive run it correctly but heres the results, note the HOST didnt support direct=1


Host
root@proxmox-02:/dev/SSD2TB# fio --ioengine=libaio --direct=0 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --size=1G --runtime=60 --time_based --name seq_read --filename=/dev/SSD2TB/test1.img seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=1388MiB/s][r=355k IOPS][eta 00m:00s] seq_read: (groupid=0, jobs=1): err= 0: pid=1341754: Tue Nov 14 21:33:02 2023 read: IOPS=338k, BW=1319MiB/s (1383MB/s)(77.3GiB/60001msec) slat (nsec): min=1293, max=261881, avg=1699.16, stdev=1074.44 clat (nsec): min=651, max=256675, avg=861.33, stdev=747.64 lat (nsec): min=1974, max=276145, avg=2560.49, stdev=1382.67 clat percentiles (nsec): | 1.00th=[ 684], 5.00th=[ 700], 10.00th=[ 708], 20.00th=[ 724], | 30.00th=[ 732], 40.00th=[ 740], 50.00th=[ 748], 60.00th=[ 820], | 70.00th=[ 844], 80.00th=[ 868], 90.00th=[ 1160], 95.00th=[ 1336], | 99.00th=[ 1528], 99.50th=[ 1656], 99.90th=[ 8640], 99.95th=[ 8768], | 99.99th=[10432] bw ( MiB/s): min= 784, max= 1473, per=100.00%, avg=1318.75, stdev=186.14, samples=119 iops : min=200942, max=377148, avg=337601.29, stdev=47652.76, samples=119 lat (nsec) : 750=49.36%, 1000=36.25% lat (usec) : 2=14.04%, 4=0.16%, 10=0.18%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=44.81%, sys=55.17%, ctx=477, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20252682,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1319MiB/s (1383MB/s), 1319MiB/s-1319MiB/s (1383MB/s-1383MB/s), io=77.3GiB (83.0GB), run=60001-60001msec


VM
docker@docker-02:/mnt$ sudo fio --ioengine=libaio --direct=0 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --size=1G --runtime=60 --time_based --name seq_read --filename=/mnt/SSD2TB/test1.img seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=368MiB/s][r=94.1k IOPS][eta 00m:00s] seq_read: (groupid=0, jobs=1): err= 0: pid=49862: Tue Nov 14 21:53:12 2023 read: IOPS=91.4k, BW=357MiB/s (374MB/s)(20.9GiB/60001msec) slat (nsec): min=1561, max=8802.3k, avg=7523.10, stdev=38347.06 clat (nsec): min=709, max=4084.8k, avg=1590.27, stdev=2523.45 lat (usec): min=2, max=8815, avg= 9.31, stdev=38.78 clat percentiles (nsec): | 1.00th=[ 1128], 5.00th=[ 1256], 10.00th=[ 1336], 20.00th=[ 1368], | 30.00th=[ 1384], 40.00th=[ 1400], 50.00th=[ 1448], 60.00th=[ 1528], | 70.00th=[ 1544], 80.00th=[ 1560], 90.00th=[ 1592], 95.00th=[ 2448], | 99.00th=[ 3568], 99.50th=[ 4832], 99.90th=[23168], 99.95th=[28288], | 99.99th=[47360] bw ( KiB/s): min=220672, max=458968, per=100.00%, avg=366331.42, stdev=57784.79, samples=119 iops : min=55168, max=114742, avg=91582.88, stdev=14446.23, samples=119 lat (nsec) : 750=0.01%, 1000=0.57% lat (usec) : 2=93.29%, 4=5.38%, 10=0.53%, 20=0.07%, 50=0.13% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 10=0.01% cpu : usr=26.65%, sys=62.23%, ctx=47213, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5485501,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=20.9GiB (22.5GB), run=60001-60001msec Disk stats (read/write): sdc: ios=86615/73, merge=0/110, ticks=60314/93, in_queue=60442, util=94.69%
 
Last edited:
LnxBil said:
You run both without direct=1, so the result are still unusable. Please run

Code: 
df /dev/SSD2TB
Click to expand...

Host

root@proxmox-02:~# df /dev/SSD2TB Filesystem 1K-blocks Used Available Use% Mounted on udev 8122664 0 8122664 0% /dev


root@proxmox-02:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K - -numjobs=1 --iodepth=1 --size=1G --runtime=60 --time_based --name seq_read --fil ename=/dev/SSD2TB/test1.img seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.33 Starting 1 process seq_read: Laying out IO file (1 file / 1024MiB) fio: looks like your file system does not support direct=1/buffered=0 fio: destination does not support O_DIRECT fio: pid=1522462, err=22/file:filesetup.c:805, func=open(/dev/SSD2TB/test1.img), error=Invalid argument Run status group 0 (all jobs):

VM

docker@docker-02:~$ df /mnt/SSD2TB Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdc 1856686824 236034864 1526263736 14% /mnt/SSD2TB

docker@docker-02:~$ sudo fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --size=1G --runtime=60 --time_based --name seq_read --filename=/mnt/SSD2TB/test1.img [sudo] password for docker: seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.28 Starting 1 process seq_read: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=23.4MiB/s][r=5991 IOPS][eta 00m:00s] seq_read: (groupid=0, jobs=1): err= 0: pid=68715: Wed Nov 15 12:25:01 2023 read: IOPS=6074, BW=23.7MiB/s (24.9MB/s)(1424MiB/60001msec) slat (usec): min=13, max=974, avg=29.45, stdev= 8.12 clat (usec): min=4, max=6397, avg=130.82, stdev=36.28 lat (usec): min=82, max=6499, avg=160.99, stdev=38.76 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 112], 10.00th=[ 113], 20.00th=[ 114], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 123], 60.00th=[ 126], | 70.00th=[ 130], 80.00th=[ 143], 90.00th=[ 159], 95.00th=[ 186], | 99.00th=[ 237], 99.50th=[ 269], 99.90th=[ 367], 99.95th=[ 408], | 99.99th=[ 906] bw ( KiB/s): min=21984, max=29106, per=100.00%, avg=24322.11, stdev=920.56, samples=119 iops : min= 5496, max= 7276, avg=6080.45, stdev=230.13, samples=119 lat (usec) : 10=0.01%, 50=0.01%, 100=0.76%, 250=98.45%, 500=0.74% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=7.34%, sys=27.73%, ctx=364512, majf=1, minf=15 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=364493,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=23.7MiB/s (24.9MB/s), 23.7MiB/s-23.7MiB/s (24.9MB/s-24.9MB/s), io=1424MiB (1493MB), run=60001-60001msec Disk stats (read/write): sdc: ios=363132/130, merge=0/136, ticks=47525/81, in_queue=47632, util=99.93%
 
As a couple of you have pointed out that QLC shouldn't be used, what types should i be looking at?
  • SLC?
  • MLC?
  • TLC?
 
Ideally anything non QLC with power loss protection (PLP) as that will push you to enterprise/datacenter SSDs. The consumer SSD market unfortunately is a bit of a minefield when you are looking for decent drives.
 
  • Like
Reactions: daNutz
aaron said:
Ideally anything non QLC with power loss protection (PLP) as that will push you to enterprise/datacenter SSDs. The consumer SSD market unfortunately is a bit of a minefield when you are looking for decent drives.
Click to expand...
ok thanks, this is for home lab and specifically for frequent data processing, so i think im likely now looking at something like the Samsung 870 EVO 2 TB.
 
Why not something like a Kingston DC600M? The price is just a bit higher but with at least a double duration.
 
daNutz said:
ok thanks, this is for home lab and specifically for frequent data processing, so i think im likely now looking at something like the Samsung 870 EVO 2 TB.
Click to expand...
Thats not what aaron recommended. Still consumer SSD without a power-loss protection (PLP) so maybe 1/100 or 1/1000 of the performance whenever you need to do sync writes. But yes, better than these QVOs. Spend a little bit more and you could buy some proper SSDs. This woud be the PM883/PM893/PM897 if you want to stick with Samsung SATA SSDs.

And you also always try to benchmark "/dev/SSD2TB/test1.img". In the "/dev" folder you find block devices but you are trying to write to a file. For that you have to mount that disk first. So you are probably just benchmarking your root filesystem and not that QVO disk when doing your host benchmarks.
 
Last edited:
  • Like
Reactions: daNutz and JensF
Dunuin said:
Thats not what aaron recommended. Still consumer SSD without a power-loss protection (PLP) so maybe 1/100 or 1/1000 of the performance whenever you need to do sync writes. But yes, better than these QVOs. Spend a little bit more and you could buy some proper SSDs. This woud be the PM883/PM893/PM897 if you want to stick with Samsung SATA SSDs.

And you also always try to benchmark "/dev/SSD2TB/test1.img". In the "/dev" folder you find block devices but you are trying to write to a file. For that you have to mount that disk first. So you are probably just benchmarking your root filesystem and not that QVO disk when doing your host benchmarks.
Click to expand...
Im aware and appreciate the recommendations but thats very overkill and expensive for my requirements. When you say spend a little more and get proper SSD's i can only see prices 3x that of what i mentioned for roughly the same size.

re the test, ive not used to this test/tool, can you provide the correct parameters to benchmark the disk both on the host and VM?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom