Slow transfer/backup speeds to external USB Hard Disk (<2MB/s)

[nothingness]

Hi All,

I have build an energy friendly server with Proxmox VE (6.2) as virtualisation platform for our home setup (Home Automatisation, Nextcloud etc).

Although, I am not a Linux specialist I have managed setting up everything so-far by following a lot of tutorials (both written en video).

Now, it comes to (external) backups I have reached a point where I am completely stuck and I could use some assistance.

The proxmox server has an SSD on board + 4 * 5TB Seagate hard disks which are placed in a ZFS pool. Through the GUI it is possible to make scheduled backups of the VM’s to a directory on the ZFS pool. This works fine!

However, I also want to make external backups and this is where the problem occurs.

As media for the external backups, I have purchased 2 * 8TB Seagate USB Hard disks (to change every other week). These have been repartitioned/reformatted with gdisk and formatted .ext4.

To auto mount the USB drives I have installed “pve6-usb-automount”.

When testing the backups (GUI setup) to the external hard disks I noticed that this was extreme slow. At first, I thought it was related to backup system itself.

As a test I tried coping a backup (from a small VM) from the zfs pool to the usb hard disk with rsync –progress and this shows a speed blow 2 MB/s!

To make sure nothing is wrong with the USB hard disks and the rest of the hardware I have created a system rescue - USB stick (system-rescue.org ) to boot from. I download a number of ISO files to test speeds with “Grsync” copying the ISO’s between the drives and the speed was > 130MB/s.

It looks like it has something to do with the setup of Proxmox VE which limits the transfer speeds to the external usb drives, but I do not have clue what is.

According to “lsusb -t” the devices are connected with 5Gb/s

I have used google a lot to find solutions, but I was not able to find a solution, yet.

Your advice/tips are appreciated!

Kind regards,
Marco

 

[wolfgang]

Hi,

you can't compare rsync with a bulk copy.
If you are interested in what your USB Disk is capable use fio[1] instead.
When you backup a CT with rsync the first time you have to read/write many small (<4KB).
This is not the best pattern for speed .

1.) https://fio.readthedocs.io/en/latest/fio_doc.html

 

[nothingness]

Hi Wolfgang,

Thanks for your reply! Tomorrow evening when I have a iittke more time, I will give fio a try.

Maybe I have a misunderstanding how Proxmox operates. But it is not only rsync. For example, I tried to upload a iso file (730mb) via the gui to the attached USB Drive and it took the drive 16 minutes to import it to the harddisk.

Will get back on the results of fio!

Best regards,
Marco

 

[nothingness]

Hi Wolfgang,

I have installed fio and run three tests arguments I found on arstechnica.

The results are as follows:

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

which results in:

Run status group 0 (all jobs):
  WRITE: bw=51.8KiB/s (53.1kB/s), 51.8KiB/s-51.8KiB/s (53.1kB/s-53.1kB/s), io=3112KiB (3187kB), run=60048-60048msec

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

which results in:

Run status group 0 (all jobs):
  WRITE: bw=5895KiB/s (6037kB/s), 368KiB/s-376KiB/s (377kB/s-385kB/s), io=367MiB (385MB), run=62632-63782msec

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

which results in

Run status group 0 (all jobs):
  WRITE: bw=20.4MiB/s (21.4MB/s), 20.4MiB/s-20.4MiB/s (21.4MB/s-21.4MB/s), io=1223MiB (1282MB), run=60042-60042msec

Does this seem right to you? It seems rather slow to me.

Thanks for your assistance!

Kind regards,
Marco

 

[wolfgang]

It looks like your USB controller is not working correctly.
To prove retest without a filesystem.
The test without fs will destroy all data on the disk.
Unmount the disk before.
You can run fio direct on the device. Also, I would use psync instead posixaio.
Not many programs use posixaio, and the backup does not.

fio --filname /dev/sd<x>

 

[nothingness]

Hi,
I have removed the partitions from the hard disk and tried the suggested change to the fio command but do not seem to get it working.

Will give it another try by tommorow.

 

[nothingness]

Hi Wolfgang,

I still struggle to run fio directly on to the unmounted/unpartitioned drive. Maybe something wrong in using the command.

However, I have been able to do another test.

I made a USB Stick with an Debian Live AMD64 GNOME ISO and started the computer running from the USB Disk (so no Proxmox VE).

When I redo the same fio tests (as above) the numbers are much higher ( >100MB).

Does this mean we could rule out any hardware issues? Or did you mean something different with the USB controller?

Kind regards,
Marco

 

[nothingness]

Any suggestions what might cause this slow transfers behaviour and solutions are more than welcome!

Would a fresh install of Proxmox VE be a valid solution?