[SOLVED] Slow Disk IO inside VM but not Proxmox

[Esity]

So I am have an interesting situation.

I have Proxmox installed on an r610 that I just purchased and have never used this software before. I can create VMs but what I noticed is they are slow. Upon further investigation it appears to be the disk.

When I log into the Proxmox box directly(not a VM) and run this DD command
"dd if=/dev/zero of=./test bs=1G count=1"
I get this output "1.8103 s, 593 MB/s" which seems fine. It's not amazing but it does the job.

Then when I create an Ubuntu VM with 2 cores, 2GB of ram with 36gb of SCSI disk from the local-lvm and run the same command, I get this
37.1261 s, 28.9MB/s

Now I understand that we are going to lose performance for multiple reasons but I cannot explain this. Input would be appreciated

 

[qubozik]

Hey!

Perhaps read this:

https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

Test different caching settings. Take caution when enabling write-back if you do not have a battery backed RAID card.

Good luck!

 

[Esity]

So I created a VM with each of the types with very little difference. However when I selected discard = true I get better speeds.

Here is another anomaly. When I use the same NFS share(1 test at a time) get different speeds.
If I create a VM with local-lvm main disk I get write speeds of 100MB/s
If I attach the NFS volume over ethernet and run a dd against it, I am getting ~100MB/s

If I create a VM with NFS main storage, I get write speeds of 10MB/s against /
If I attach that same NFS volume, I get 18MB/s

If I create a VM with iSCSI storage(same NAS server, dedicated disk) I get 23MB/s
if I attach the same NFS volume as before, I get ~100MB/s


Why does this make any sense?

 

[udo]

When I log into the Proxmox box directly(not a VM) and run this DD command
"dd if=/dev/zero of=./test bs=1G count=1"
I get this output "1.8103 s, 593 MB/s" which seems fine. It's not amazing but it does the job.

Hi,
sorry but your dd-test say nothing. You couldn't say how much data arived the disk in the 1.8 second. Only that you are able to copy 1GB in 1.8 second to the cache (ram).
You must sync, and - if your underlying filesystem is zfs - writing zeros are useless, because compression.

if you don't use zfs the dd-switch "conv=fdatasync" will be helpfull but better is to use real performance test programs like fio (but the result is not so easy to read )

Udo

 

[Esity]

I figured it out. Between Ubuntu and Proxmox it was created a swap drive automatically. Disabled swap and I am getting close to 110MB/s across the board now. Thanks for the help

 

[Dimitri]

I figured it out. Between Ubuntu and Proxmox it was created a swap drive automatically. Disabled swap and I am getting close to 110MB/s across the board now. Thanks for the help

I'm facing the same issue. Could you please explain how to disable swap ?

Any guidelines will be really helpfull

 

[Esity]

if you are running ubuntu you can run
"sudo swap off -a" however this will be removed by a reboot.
Look into swapiness settings for more info.
https://unix.stackexchange.com/ques...nfigure-swappiness-in-linux-memory-management

 

[spirit]

I'm facing the same issue. Could you please explain how to disable swap ?

Any guidelines will be really helpfull

#swapoff -a , to disable it online

and remove swap line in /etc/fstab to have it persistent across reboot.

 

[SamTzu]

Hi,
sorry but your dd-test say nothing. You couldn't say how much data arived the disk in the 1.8 second. Only that you are able to copy 1GB in 1.8 second to the cache (ram).
You must sync, and - if your underlying filesystem is zfs - writing zeros are useless, because compression.

if you don't use zfs the dd-switch "conv=fdatasync" will be helpfull but better is to use real performance test programs like fio (but the result is not so easy to read )

Udo

For my self... I never use compression on KVM's even when they are ZFS. It's easy to disable when you build your KVM Proxmox host. Disk space is cheap. Currently our bottle neck is RAM (because ZFS eats like it's Christmas.) Lack of RAM often causes IO slow down on VM's.