slow file transfer

[Covy]

I set up my first VM (Windows 2019). I'm trying to transfer a file that is 81GB from my file server (UNRAID). When I transfer it from UNRAID to my local machine, the transfer rate is about 110-120 MB/S. However, when I am in my Windows 2019 proxmox VM, the transfer rate is about 10-20 MB/s.



It starts off normal, then quickly tapers off. I have it setup like this:





I'm thinking that being new, I just don't have something setup properly - but I haven't found it yet and searched a bit trying to figure out why this is happening. I changed from E1000 to VirtIO to see if that would help (based upon some forum threads). But - that was a short-lived increase at the beginning of the file transfer.

**EDIT** I forgot to mention that I also have the VirtIO drivers installed on the Windows 2019 vm.

Thanks in advance!

 

[Chris]

I set up my first VM (Windows 2019). I'm trying to transfer a file that is 81GB from my file server (UNRAID). When I transfer it from UNRAID to my local machine, the transfer rate is about 110-120 MB/S. However, when I am in my Windows 2019 proxmox VM, the transfer rate is about 10-20 MB/s.

View attachment 47417

It starts off normal, then quickly tapers off. I have it setup like this:

View attachment 47418

View attachment 47419

I'm thinking that being new, I just don't have something setup properly - but I haven't found it yet and searched a bit trying to figure out why this is happening. I changed from E1000 to VirtIO to see if that would help (based upon some forum threads). But - that was a short-lived increase at the beginning of the file transfer.

**EDIT** I forgot to mention that I also have the VirtIO drivers installed on the Windows 2019 vm.

Thanks in advance!

Hi,
what storage are you using for your VM disk and especially, what physical disks are you using for that storage?

 

[Covy]

Hi,
what storage are you using for your VM disk and especially, what physical disks are you using for that storage?

sda = location of VM Disk for OS
sdb = location of the VM Disk that I am copying data to

Both are SSD. sda = WD Blue, sdb = Crucial

 

[Chris]

View attachment 47420

sda = location of VM Disk for OS
sdb = location of the VM Disk that I am copying data to

Both are SSD. sda = WD Blue, sdb = Crucial

Have you enabled trim support for the disk? Please post the output of qm config <VMID> as code block, replacing VMID with the id of your VM

 

[Covy]

Have you enabled trim support for the disk? Please post the output of qm config <VMID> as code block, replacing VMID with the id of your VM

I probably haven't enabled that since I don't know what it is or where to enable it

root@pve:~# qm config 100
boot: order=ide2;net0
cores: 3
ide0: local-lvm:vm-100-disk-1,size=80G
ide1: SSD2:100/vm-100-disk-0.qcow2,size=1500G
ide2: local:iso/en_windows_server_2019_x64_dvd_4cb967d8.iso,media=cdrom,size=4729754K
machine: pc-i440fx-7.1
memory: 16480
meta: creation-qemu=7.1.0,ctime=1676859531
name: Windows2019
net0: virtio=DE:AC:2D:E2:89:AC,bridge=vmbr0
numa: 0
ostype: win10
parent: SnapshotSQLInstalled
scsihw: virtio-scsi-single
smbios1: uuid=19aa0d7b-8285-4a06-8e94-128e61e742c2
sockets: 4
unused0: local-lvm:vm-100-disk-0
vga: qxl
vmgenid: f6248ca7-2808-494b-8b0c-017e655b75fc

 

[Dunuin]

That drop in performance usually should be because caching. First it's fast because it is writing to cache, then the cache gets full and you will see the real TLC/QLC NAND performance. So I would try to benchmark your VMs disk. If writing there is slow in general when writing some dozens of GBs, then you know its the VMs storage and not a network problem or the remote storage performance.

 

[Covy]

That drop in performance usually should be because caching. First it's fast because it is writing to cache, then the cache gets full and you will see the real TLC/QLC NAND performance. So I would try to benchmark your VMs disk. If writing there is slow in general when writing some dozens of GBs, then you know its the VMs storage and not a network problem or the remote storage performance.

I'm relatively sure it isn't network or remote storage because I can copy the same file to my local SSD (WD Blue) on my PC and it is a sustained 100-120MB, which I think is around max for SMB anyway.

 

[Chris]

I'm relatively sure it isn't network or remote storage because I can copy the same file to my local SSD (WD Blue) on my PC and it is a sustained 100-120MB, which I think is around max for SMB anyway.

So the slow speed is probably because of the drives cache being full and you get the "true" TLC speed as mentioned by @Dunuin .

Nevertheless a few recommendations for your setup:

• Install and enable the qemu guest agent, by going to the VMs Options tab
• Install the latest virtio drivers (which you mentioned you already did) and switch your disks from ide to virtio-scsi
• Enable Discard and SSD Emulation for the disks by going to the VMs Hardware tab, selecting the disk and clicking on Edit

 

[Covy]

So the slow speed is probably because of the drives cache being full and you get the "true" TLC speed as mentioned by @Dunuin .

Nevertheless a few recommendations for your setup:

• Install and enable the qemu guest agent, by going to the VMs Options tab
• Install the latest virtio drivers (which you mentioned you already did) and switch your disks from ide to virtio-scsi
• Enable Discard and SSD Emulation for the disks by going to the VMs Hardware tab, selecting the disk and clicking on Edit

ok, I've done that. will have to restart.

However - I've been copying 600GB worth of files for over 24 hours now. Still says 'more than 1 day left'. I've never has my ESXi images with similar drives copy these same files for more than 5 hours. I don't think it is the drives or networks or ESXi wouldn't have been able to do it.

For notes... I'm running an installation that relies on a SQL Server database and upgrading that database fileset. The database files are 364GB (mdf) and 259GB (ldf). Each time I run the installation I have to move the old files into the server to test the upgrade process. I've had to do this process about 10 times, so far. I decided to give proxmox a try last week just to have fun and stood up this Windows 2019 server on it. I expect to run this installation a few more times to get the steps figured out and solidified, but at this rate I can't do it with a Proxmox VM. I lost a full day copying files and it still isn't even close to being completed

 

[Covy]

Been hovering between 0 KB/s and about a max of 2.5 MB/s.

 

[Covy]

Another note... I have transferred these same files to this same VM once previously. This is the 2nd test. I had forgotten about that until I read through my notes. Sorry, it has been a long few weeks with this crappy process.

I do not remember it taking this long the first time, so it must have been a fairly normal transfer.

So - I guess this just clouds the issue even more, unfortunately.

 

[Covy]

Made those changes and just rebooted regardless of it being in the middle of the file transfer - speed is now averaging 80 MB/s. So - looks like both files will complete in close to 2 hours. So - not sure why these changes would do it?

• Enable Discard and SSD Emulation for the disks by going to the VMs Hardware tab, selecting the disk and clicking on Edit

Those were the only updates I made before the reboot.

 

[Chris]

Made those changes and just rebooted regardless of it being in the middle of the file transfer - speed is now averaging 80 MB/s. So - looks like both files will complete in close to 2 hours. So - not sure why these changes would do it?

• Enable Discard and SSD Emulation for the disks by going to the VMs Hardware tab, selecting the disk and clicking on Edit

Those were the only updates I made before the reboot.

Discard enables trim support for storages which support it. This can make a huge difference, since the SSD controller can write to clear blocks without having to read them first.

 

[Covy]

Discard enables trim support for storages which support it. This can make a huge difference, since the SSD controller can write to clear blocks without having to read them first.

Gotcha. Did not know this. Does ESXi do this by design or something instead of optional? I've just never seen this with VMs before, but I've been using that for the last 15 years so I really have no experience with other hypervisors.

 

[Chris]

Does ESXi do this by design or something instead of optional?

I have never touched ESXi myself, so I cannot comment on that.

 

[Covy]

Unrelated, but I noticed that with the Spice client - OBS (Open Broadcaster Software) cannot capture the video/image of that window.