Proxmox slow file transfer performance

[shtorm93]

Hello.
I have multiple VM up and running, but have some weird problems with file transfer performance.
VMs are on a NVMe drive, iperf between VMs shows a result of 22GB/s, and dhparam shows almost full speed of the NVMe drive (over 1 gb/s). Both drive and network ar VirtIO. Both VMs running Ubuntu 18. I did migrate these vm from a different hypervisor, bet even creating new VMs gets me same results. WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.
Problem is when i try to use WinSCP, Filezilla or web based file transfer. The speed never goes past 50mb/s, sometimes even lower.

I'm not sure how to narrow down the problem. Any advice appreciated.
Thank you.

 

[oguz]

hi,

WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.
Problem is when i try to use WinSCP, Filezilla or web based file transfer. The speed never goes past 50mb/s, sometimes even lower.

which physical machine? is it on the same network as the PVE?

 

[shtorm93]

hi,




which physical machine? is it on the same network as the PVE?

It is on the same network with the same subnet mask, connected with 10G switch. Uploading an .ISO file works fine and at full speed. But WinSCP copy gets a maximum of 50mb/s, sometimes less.

more context:
I made a RAMdisk on both a virtual windows machine in PVE and a physical one. File transfers reached almost full 10G speeds.

Made a new test VM using ubuntu server, WinSCP again cannot pass 50mb/s, even tho iperf shows insane 22G speeds between VMs.

 

[shtorm93]

My specific issue is - I have a file synch server, and transfer speeds are low. So troubleshooting this problem i was trying FTP and WinSCP and this got me similar results. But testing drive speed and network separately reveal no issue and high speeds. The slow WinSCP transfer to PVE host got me even more confused since as i mentioned before I did not have any trouble transferring .ISO at full speed.

 

[oguz]

could the bottleneck be caused by winscp? do you get it with something else?

 

[shtorm93]

could the bottleneck be caused by winscp? do you get it with something else?

I used Filezilla to try and copy files through FTP. same results. I'm running owncloud on one of the VMs, and that is what is not working at the expected speeds, so for testing purposes I used WinSCP and filezilla.
I tried to search if there is anything on the VM os that could limit the transfer speed, but found nothing. and again- the confusing part is that WinSCP copy to PVE is at low speed, so i cannot pinpoint if the problem is my network, disk settings, VM os settings or PVE network. I also tried to turn off firewall on PVE.

btw, thank you for your time.

 

[oguz]

one way to narrow it down is running atop on the PVE host while doing the big file transfer. that way you can see if it's a disk or network related problem

 

[shtorm93]

sda ad sdb are 2 SSDs in raid, they hover around 40%. enp6s0 is barely used at 4%. Pictures 1 and 2.

scp shows actual SSD speeds and atop shows 90% busy. Picture 4 and 5.

if i would just need to copy a file I would use scp and leave it at that, but it seams that the problem that limits transfer speeds on WinSCP and Filezilla (both to PVE and to VM) also does the same with my owncloud server while synchronizing files, at least i come to this conclusion since the speed is similar. Also, PVE is instaled on SATA SSDs and my VM is on a NVMe SSD, so I'm getting even smaller fraction of the speed i should.

Also, I tried to increase both the CPU and RAM resources for the VM, but it had little to no impact on transfer speeds.

Any advice on how to pinpoint this problem.

 

[focusedinternet]

For what it's worth...

I was experiencing the same exact issues with network transfer speed as well. I too was copying files from my PC to my PVE server, with WinSCP, and was only getting about 41-45MB/sec on a gigabit link. Atop was showing about ~450Mbps when copying files with WinSCP. I couldn't figure out what was going on so I started poking around here on the forums. I was about to set up iPerf when I thought of just trying to use the ISO upload feature on my PVE server. So, I uploaded the Windows10-1909.iso and Ubuntu 18.04.4-server.iso. Atop showed network speeds topping 950Mbps (full gigabit thoughput). I made zero adjustments to my system. The commonality seems to be WinSCP file transfer. So my question is why transfer speeds are so different? It obviously isn't the hardware but seems to be protocol driven(?).

I'd be very interested in any thoughts you might have.

Thanks very much!

-Rob

 

[oguz]

The commonality seems to be WinSCP file transfer. So my question is why transfer speeds are so different? It obviously isn't the hardware but seems to be protocol driven(?).

might be that winscp has some builting bandwidth limit, i'm not too familiar with that software but it wouldn't hurt to check the settings or ask the maintainer about it

in the case that it's protocol driven, you'd have the same problem while transfering with the regular scp tool

 

[ChristoperLozano]

WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.

 

[oguz]

WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.

i guess we can safely say it's a winscp issue then

 

[shtorm93]

i guess we can safely say it's a winscp issue then

Not sure. FTP transfer has similar limit (using filezilla). I'm getting around 30mb/s.
And again, that is not the main problem. The problem is that one of the virtual machines running owncloud gets similar slow speeds. Did not have this problem on vmware esxi.

I will try some other options. so far i tried getting a fresh Nextcloud install and I got good read speeds but write speeds still way too low, well below hHD speeds even tho the VM is on a nvme SSD.

 

[Appollonius]

Funny, I have a similar problem... I try to copy files from my desktop to my plex server vm in ProxMox with WinSCP but have 10MB/s on a 1gbit connection to my server in a datacenter... It is weird as Filezilla and Cyberduck show the same problem... Any solutions on this?

 

[alfhabetagama]

I have the same problem, when transfering using FileZilla / Winscp it can only be in the range of 30 / mbps. Then try copying it directly from an external HDD (USB 3.0), copying around 1 TB about 6 hours, is there something wrong with my hardware?

Is there another way to copy / transfer files faster?

 

[Marvin]

I am having the same issue but only with a Windows 10 guest (Windows 7 guests works normally >300Mb/sec). Any time i try transferring a large file (10Mb) with FTP, WinSCP, through RDP or VNC the transfer shows small burst of < 50kb/sec and eventually times out. NIC drivers are latest virtio. Win10 best practices have been followed.

 

[jieiku]

I am currently investigating this on my system. I am using Ubuntu 20.04 with plasma desktop.

What I have found is that if I use dolphins built in sftp functionality then files transfer 35-50 MiB/s.
If I mount the share in fstab using cifs OR use rsync then I get full speed.

rsync -avxe ssh /home/jieiku/_iso/kubuntu-20.04-desktop-amd64.iso pve:/mnt/pve/stuff/ --info=progress2

The info I am currently finding is that the program your using to copy with needs to be asynchronous to get full speed.
I also found it suggested that nautilus will transfer at full speed unlike dolphin, so I tried it and it does indeed transfer at full speed.

@Marvin your issue sounds a bit different than the OP's I would guess its not the same issue.

 

[Indigo]

Had the same problem, 1 Gbps connection and transfer with 45-40MBps
Go to Advance Site Settings -> Connection -> "Uncheck" Optimize connection buffer size
now transfer with 80 MBps

 

[puzon]

Similar problem here i think - Windows VM with two disk on ceph storage - all connected on 10G. Transfer between two disk - about 40Mb/s

 

[IcarusFalling]

+1 for the same problem.
WinSCP on windows 10 VM to PVE host on the same bridge, windows NIC says 10GB speed --> 35MBs

FileZilla ubuntu 22.04 VM to PVE host on the same bridge, --> 65 MBs

windows 10 VM copying from a truenas VM on the same bridge --> 400MBs

WinSCP from windows 10 VM to the ubuntu VM --> 16MBs!

atop showing the same behaviour of nvme on both windows and ubuntu around 90% busy