Proxmox slow file transfer performance

shtorm93

New Member
Feb 4, 2020
6
0
1
29
Hello.
I have multiple VM up and running, but have some weird problems with file transfer performance.
VMs are on a NVMe drive, iperf between VMs shows a result of 22GB/s, and dhparam shows almost full speed of the NVMe drive (over 1 gb/s). Both drive and network ar VirtIO. Both VMs running Ubuntu 18. I did migrate these vm from a different hypervisor, bet even creating new VMs gets me same results. WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.
Problem is when i try to use WinSCP, Filezilla or web based file transfer. The speed never goes past 50mb/s, sometimes even lower.

I'm not sure how to narrow down the problem. Any advice appreciated.
Thank you.
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
4,849
608
118
hi,


WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.
Problem is when i try to use WinSCP, Filezilla or web based file transfer. The speed never goes past 50mb/s, sometimes even lower.

which physical machine? is it on the same network as the PVE?
 

shtorm93

New Member
Feb 4, 2020
6
0
1
29
hi,




which physical machine? is it on the same network as the PVE?

It is on the same network with the same subnet mask, connected with 10G switch. Uploading an .ISO file works fine and at full speed. But WinSCP copy gets a maximum of 50mb/s, sometimes less.

more context:
I made a RAMdisk on both a virtual windows machine in PVE and a physical one. File transfers reached almost full 10G speeds.

Made a new test VM using ubuntu server, WinSCP again cannot pass 50mb/s, even tho iperf shows insane 22G speeds between VMs.
 

shtorm93

New Member
Feb 4, 2020
6
0
1
29
My specific issue is - I have a file synch server, and transfer speeds are low. So troubleshooting this problem i was trying FTP and WinSCP and this got me similar results. But testing drive speed and network separately reveal no issue and high speeds. The slow WinSCP transfer to PVE host got me even more confused since as i mentioned before I did not have any trouble transferring .ISO at full speed.
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
4,849
608
118
could the bottleneck be caused by winscp? do you get it with something else?
 

shtorm93

New Member
Feb 4, 2020
6
0
1
29
could the bottleneck be caused by winscp? do you get it with something else?

I used Filezilla to try and copy files through FTP. same results. I'm running owncloud on one of the VMs, and that is what is not working at the expected speeds, so for testing purposes I used WinSCP and filezilla.
I tried to search if there is anything on the VM os that could limit the transfer speed, but found nothing. and again- the confusing part is that WinSCP copy to PVE is at low speed, so i cannot pinpoint if the problem is my network, disk settings, VM os settings or PVE network. I also tried to turn off firewall on PVE.

btw, thank you for your time.
 
Last edited:

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
4,849
608
118
one way to narrow it down is running atop on the PVE host while doing the big file transfer. that way you can see if it's a disk or network related problem
 
  • Like
Reactions: node1 and Moxxorp

shtorm93

New Member
Feb 4, 2020
6
0
1
29
sda ad sdb are 2 SSDs in raid, they hover around 40%. enp6s0 is barely used at 4%. Pictures 1 and 2.

scp shows actual SSD speeds and atop shows 90% busy. Picture 4 and 5.

if i would just need to copy a file I would use scp and leave it at that, but it seams that the problem that limits transfer speeds on WinSCP and Filezilla (both to PVE and to VM) also does the same with my owncloud server while synchronizing files, at least i come to this conclusion since the speed is similar. Also, PVE is instaled on SATA SSDs and my VM is on a NVMe SSD, so I'm getting even smaller fraction of the speed i should.

Also, I tried to increase both the CPU and RAM resources for the VM, but it had little to no impact on transfer speeds.

Any advice on how to pinpoint this problem.
 

Attachments

  • 1.png
    1.png
    7.1 KB · Views: 75
  • 2.png
    2.png
    105.3 KB · Views: 75
  • 4.png
    4.png
    14.4 KB · Views: 57
  • 5.png
    5.png
    106.2 KB · Views: 63

focusedinternet

New Member
Feb 19, 2020
2
0
1
51
For what it's worth...

I was experiencing the same exact issues with network transfer speed as well. I too was copying files from my PC to my PVE server, with WinSCP, and was only getting about 41-45MB/sec on a gigabit link. Atop was showing about ~450Mbps when copying files with WinSCP. I couldn't figure out what was going on so I started poking around here on the forums. I was about to set up iPerf when I thought of just trying to use the ISO upload feature on my PVE server. So, I uploaded the Windows10-1909.iso and Ubuntu 18.04.4-server.iso. Atop showed network speeds topping 950Mbps (full gigabit thoughput). I made zero adjustments to my system. The commonality seems to be WinSCP file transfer. So my question is why transfer speeds are so different? It obviously isn't the hardware but seems to be protocol driven(?).

I'd be very interested in any thoughts you might have.

Thanks very much!

-Rob
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
4,849
608
118
The commonality seems to be WinSCP file transfer. So my question is why transfer speeds are so different? It obviously isn't the hardware but seems to be protocol driven(?).

might be that winscp has some builting bandwidth limit, i'm not too familiar with that software but it wouldn't hurt to check the settings or ask the maintainer about it

in the case that it's protocol driven, you'd have the same problem while transfering with the regular scp tool
 

ChristoperLozano

New Member
Feb 22, 2020
1
0
1
31
WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
4,849
608
118
WinSCP connection from a physical machine to proxmox server also has the same speed limit, even tho copying files from an old server via command line had the full drive speed performance.

i guess we can safely say it's a winscp issue then
 

shtorm93

New Member
Feb 4, 2020
6
0
1
29
i guess we can safely say it's a winscp issue then
Not sure. FTP transfer has similar limit (using filezilla). I'm getting around 30mb/s.
And again, that is not the main problem. The problem is that one of the virtual machines running owncloud gets similar slow speeds. Did not have this problem on vmware esxi.

I will try some other options. so far i tried getting a fresh Nextcloud install and I got good read speeds but write speeds still way too low, well below hHD speeds even tho the VM is on a nvme SSD.
 

Appollonius

Member
Feb 20, 2020
36
3
8
24
Funny, I have a similar problem... I try to copy files from my desktop to my plex server vm in ProxMox with WinSCP but have 10MB/s on a 1gbit connection to my server in a datacenter... It is weird as Filezilla and Cyberduck show the same problem... Any solutions on this?
 

alfhabetagama

New Member
Apr 22, 2020
1
0
1
38
I have the same problem, when transfering using FileZilla / Winscp it can only be in the range of 30 / mbps. Then try copying it directly from an external HDD (USB 3.0), copying around 1 TB about 6 hours, is there something wrong with my hardware?

Is there another way to copy / transfer files faster?
 

Marvin

Active Member
Jan 20, 2011
40
1
28
I am having the same issue but only with a Windows 10 guest (Windows 7 guests works normally >300Mb/sec). Any time i try transferring a large file (10Mb) with FTP, WinSCP, through RDP or VNC the transfer shows small burst of < 50kb/sec and eventually times out. NIC drivers are latest virtio. Win10 best practices have been followed.
 

jieiku

Member
Jan 1, 2020
24
14
8
41
I am currently investigating this on my system. I am using Ubuntu 20.04 with plasma desktop.

What I have found is that if I use dolphins built in sftp functionality then files transfer 35-50 MiB/s.
If I mount the share in fstab using cifs OR use rsync then I get full speed.

Code:
rsync -avxe ssh /home/jieiku/_iso/kubuntu-20.04-desktop-amd64.iso pve:/mnt/pve/stuff/ --info=progress2

The info I am currently finding is that the program your using to copy with needs to be asynchronous to get full speed.
I also found it suggested that nautilus will transfer at full speed unlike dolphin, so I tried it and it does indeed transfer at full speed.

@Marvin your issue sounds a bit different than the OP's I would guess its not the same issue.
 

Indigo

New Member
Dec 14, 2020
1
1
3
33
Had the same problem, 1 Gbps connection and transfer with 45-40MBps
Go to Advance Site Settings -> Connection -> "Uncheck" Optimize connection buffer size
now transfer with 80 MBps
 
  • Like
Reactions: SamirD

puzon

New Member
Jun 16, 2020
19
0
1
44
Similar problem here i think - Windows VM with two disk on ceph storage - all connected on 10G. Transfer between two disk - about 40Mb/s
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!