slow transfer speeds to ext HDD with direct passthrough

[castielsn]

Hello Everyone!

I set up an openmediavault 6 VM in proxmox and passed an external HDD (brand new, tested on other system, full read/write speeds) through to it via the usb device passthrough option in the web UI, I passed through the full port. OMV sees the disk without any problem, I set up a share with SMB. The problem is, that the transfer speeds are extremely slow, about 4 (!) MB/s.

I tried to pass through another usb 3 flash drive, same slow speed.

EDIT: I installed proxmox on a different machine, created an OMV guest OS, then I created a virtual HDD and added it to the OMV guest as well, created an SMB share there too, and again, same extremely slow 4 MB˛/s. Tried the ext. HDD with passthgough again, same slow speed. What is the bottleneck here? However, when I installed OMV on the same machine, bare metal, without proxmox, I got full 1 gigabit speed on the ext. HDD.

Is there something I'm missing here? I was expecting to have about a 100 MB/s speeds on my gigabit network in the same way as I have with my OMV NAS installed on bare metal. Any suggestions?

Thank you!

 

[shanreich]

Did you pass through the whole controller or just one USB device? If you're not passing the whole USB controller then bad performance is to be expected, since the USB controller has to be emulated in software. You can pass through the whole USB controller via PCIe passthrough [0]

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_passthrough

 

[castielsn]

Whole controller was passed through. I tested speed without ext. HDD, with internal virtual HD (SSD), same extremely slow speeds, 4 MB/s. So it's not a passthrough issue. Installed PVE on another serves, same speeds, so its not the hardware. Home network is gigabit, everything else works full gigabit in my network, bare metal OMV NAS included. Checked lan cables, no issues there. Even the upload of an iso is slow. Everything is basically default setup in proxmox. Network is in default bridge. So now its happening in two different servers, both capable of full gigabit, both new PVE installs. Everything is up to date, too. Extremely strange.

 

[shanreich]

Okay I was just wondering since you wrote that you used the USB Device Passthrough option from the Web UI, which would be different from the PCI Device option.

Since you already tried using a virtual HD that uses the internal HD, can you just quickly verify the writing speed when writing to the device from the VM? This way we could rule out some stuff for sure.

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync

EDIT:
Can you also attach your configuration for the VM in question?

 

[castielsn]

Hi Shanreich! Thank you for the reply. The dd code returned the following:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.96246 s, 135 MB/s

 

[castielsn]

Okay, so now I am fully confused. Previously, I have restarted all machines in questions multiple times. Now, after running the dd code in the previous post, the file transfer speeds are full gigabit. ?? What happened here? I presume that this dd code is just running some test, doesn't it? So it does not actually affect (or at least should not) the file transfer speeds between the VM and my home network. How come now I have normal, full gigabit speeds from the VM to my home network and vice versa? In any case, thank you for the help, although I would really appreciate if anyone could explain to me what might have happened here.

 

[shanreich]

Okay, so now I am fully confused. Previously, I have restarted all machines in questions multiple times. Now, after running the dd code in the previous post, the file transfer speeds are full gigabit. ?? What happened here? I presume that this dd code is just running some test, doesn't it? So it does not actually affect (or at least should not) the file transfer speeds between the VM and my home network. How come now I have normal, full gigabit speeds from the VM to my home network and vice versa? In any case, thank you for the help, although I would really appreciate if anyone could explain to me what might have happened here.

You mean that you had the slow speeds until you ran the dd command, then your OMV share runs at full speed? That seems really weird indeed, I don't have an immediate answer for that.

 

[castielsn]

You mean that you had the slow speeds until you ran the dd command, then your OMV share runs at full speed? That seems really weird indeed, I don't have an immediate answer for that.

Exactly. Which is why I'm totally confused now. I actually can't replicate the slow speed problem on the pve server that I ran the dd command on, so what I did was that I reinstalled pve on the other server, set up an OMV VM, set up a share with an internal virtual HD and when I tried to copy files, same slow speeds happened. Just to clarify, on pve server 1, speeds are now absolutely fine, on pve server 2, speeds are at 4 MB/s. At this point I think I can safely say that it is definitely not hardware related, at least as far as the servers are concerned. BTW, I passed through the ext. HDD to pve server 1 and created an SMB share on it, and full gigabit speeds, so the ext. HDD is also working properly. If I pass through the same HDD to the OMV VM on pve server 2, I get the 4 MB/s speeds. So now that I was able to replicate the issue on another server, I tried the dd command on that server (pve server 2) too, and this is what it resulted in:

oot@proxmox2test:~# dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.05711 s, 351 MB/s

What in gods name is going on here? Same local network, same switch, router, two fresh proxmox and OMV VM installs and in one case full gigabit speeds and in the other only 4 MB/s.

 

[castielsn]

And the "worst" is that on pve server 1, initially I had the same slow speeds, and one time it just went full speed, and I can't find out what made it go from 4 MB/s to full gigabit...

 

[leesteken]

Maybe the HDD uses SMR and the append area got full from all the writes (and therefore it got really slow) until the drive reorganized the data in the background (a bit like an SSD)?

 

[castielsn]

The external HDD in question is brand new, and since I tried with other ext. HDD's, fast usb flash drives as well as internal virtual HDs (and remember, the issue first happened on two different servers with two different internal SSDs), I think it is safe to say that this must be something other than hardware.

On other places I read that there is an iperf test with which it is possible to check network speeds. I think I'm going to try to do this test and see what it shows.

Also, since it seems that the slow speed has nothing really to do with neither passthrough or an ext. HDD, should I rename the thread? What is common practice in such a case? Thank you!