[SOLVED] Slow performance on USB ports

[jonasl]

I'm having really poor performance when copying data from one physical USB-C disk to another physical USB-A disk plugged into the same Proxmox VE-system.

The rsync below is only at 0.85kB/s. Filesystems are both ext4 and the disks are created as VM-disks: directories and raw.

In the VM doing the copying using rsync:

/mnt# rsync -a --info=progress2 . /mnt2/
      7,866,013   0%    0.85kB/s    2:29:38 (xfr#56, ir-chk=1000/10199970)

And on the physical system:

root@proxmox:~# mount|grep sdb
/dev/sdb1 on /mnt/data type ext4 (rw,relatime)
root@proxmox:~# mount|grep sdc
/dev/sdc on /mnt/data2 type ext4 (rw,relatime)
root@proxmox:~#

 

[fabian]

how did you pass the disks to the VM? please include pveversion -v output and the relevant config files (guest, storage, ..)

 

[SOLTECSIS - Carles Munyoz]

What is the output of the `iotop` command while you are running the rsync?

 

[jonasl]

Iotop output below and pveversion output has been attached. Thanks for a quick reply!

# iotop --batch -t -o -n 5
13:15:32 Total DISK READ:         4.29 M/s | Total DISK WRITE:         0.00 B/s
13:15:32 Current DISK READ:       4.29 M/s | Current DISK WRITE:       0.00 B/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
13:15:32 17145 be/4 root        4.29 M/s    0.00 B/s  0.00 % 54.54 % rsync -a --info=progress2 . /mnt2/
13:15:33 Total DISK READ:         2.50 M/s | Total DISK WRITE:         0.00 B/s
13:15:33 Current DISK READ:       2.50 M/s | Current DISK WRITE:       0.00 B/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
13:15:33 17146 be/4 root     1270.12 K/s    0.00 B/s  0.00 % 82.42 % rsync -a --info=progress2 . /mnt2/
13:15:33 17145 be/4 root     1289.96 K/s    0.00 B/s  0.00 % 14.40 % rsync -a --info=progress2 . /mnt2/
13:15:33 14717 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.01 % [kworker/3:2-events_freezable_power_]
13:15:34 Total DISK READ:         3.63 M/s | Total DISK WRITE:         0.00 B/s
13:15:34 Current DISK READ:       3.63 M/s | Current DISK WRITE:       0.00 B/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
13:15:34 17146 be/4 root      666.35 K/s    0.00 B/s  0.00 % 73.78 % rsync -a --info=progress2 . /mnt2/
13:15:34 17145 be/4 root        2.98 M/s    0.00 B/s  0.00 % 22.80 % rsync -a --info=progress2 . /mnt2/
13:15:35 Total DISK READ:      1047.42 K/s | Total DISK WRITE:         0.00 B/s
13:15:35 Current DISK READ:    1047.42 K/s | Current DISK WRITE:       0.00 B/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
13:15:35 17146 be/4 root     1047.42 K/s    0.00 B/s  0.00 % 99.10 % rsync -a --info=progress2 . /mnt2/
13:15:35 14717 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.01 % [kworker/3:2-events_freezable_power_]
13:15:36 Total DISK READ:       555.26 K/s | Total DISK WRITE:         0.00 B/s
13:15:36 Current DISK READ:     555.26 K/s | Current DISK WRITE:       0.00 B/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
13:15:36 17146 be/4 root      555.26 K/s    0.00 B/s  0.00 % 98.92 % rsync -a --info=progress2 . /mnt2/

 

[jonasl]

qemu node config file:

root@proxmox:/etc/pve# more ./nodes/proxmox/qemu-server/108.conf
agent: 1
boot: order=scsi0;ide2;net0
cores: 1
ide2: local:iso/debian-live-10.8.0-amd64-standard.iso,media=cdrom,size=945M
memory: 4096
name: plugin-mirror
net0: virtio=F2:92:A5:54:1E:16,bridge=vmbr0
net1: virtio=CA:2A:F7:6F:C4:1B,bridge=vmbr2
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-108-disk-0,size=16G
scsi1: storageprox2:108/vm-108-disk-0.raw,size=900G
scsi2: storageprox:108/vm-108-disk-1.raw,size=3800G
scsihw: virtio-scsi-pci
smbios1: uuid=5a9d5186-4413-4e9a-acc5-8b9770358289
sockets: 4
unused0: storageprox:108/vm-108-disk-0.raw
vmgenid: 3b5486b6-c0f3-44c1-80f7-13c1e84b4604

 

[jonasl]

And storage.cfg:

dir: storageprox
	path /mnt/data
	content images
	prune-backups keep-all=1
	shared 1

dir: storageprox2
	path /mnt/data2
	content images
	prune-backups keep-all=1
	shared 0

 

[jonasl]

Hmm, i did some testing and writing 100mb file directly on the physical host to the disks are just fine. The performance inside the VM is also very good.

So the issue here seems to be related to rsync

 

[fabian]

if you monitor your disks with iostat you'll see more relevant metrics. likely you are hitting the iops bottleneck (writing a single large file vs doing lots of smaller reads/writes is a very different workload!)

 

[SOLTECSIS - Carles Munyoz]

What about the USB ports in use?
Are both disks connected to USB 3 ports?

 

[jonasl]

There is something fishy with the USB-drive. Running two speed-tests with hdparam results in different speeds:

root@proxmox:~# sudo hdparm -tT /dev/sdb1

/dev/sdb1:
 Timing cached reads:     2 MB in  3.13 seconds = 653.49 kB/sec
 Timing buffered disk reads: 164 MB in  3.06 seconds =  53.55 MB/sec
root@proxmox:~# sudo hdparm -tT /dev/sdb1

/dev/sdb1:
 Timing cached reads:   33858 MB in  1.99 seconds = 17048.15 MB/sec
 Timing buffered disk reads:  10 MB in  3.06 seconds =   3.27 MB/sec
root@proxmox:~#

 

[SOLTECSIS - Carles Munyoz]

To rule out that the problem is in the USB disks, can you make the same test connecting the USB disks into another computer?