Move storage option to single sata disk very slow < 5 MB/s

kshesq

Active Member
May 7, 2020
40
1
28
47
Hi, I have a 2 node cluster that is drbd connected. I want to offline transfer an old raw disk to a disk on the same node but this process is very slow. I only get around 2.8MB/s making a 192G disk move a tedious task.
Also I see >15% IO delay on the node when the qemu-img command is running.

The move I am doing is from a hardware RAID1 with 2 HGST WD Ultrastar HUS726T4TALE6L4 4TB 7200 RPM 512e SATA 6Gb/s to a single HGST WD Ultrastar HUS726T4TALE6L4 4TB 7200 RPM 512e SATA 6Gb/s

The RAID controller is a LSI MegaRAID 9341-4i

I also tried the move with all guests off but this did not make a difference.

specs.JPG

Package versions
Code:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-7
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-18-pve: 6.2.16-18
pve-kernel-5.15.126-1-pve: 5.15.126-1
pve-kernel-5.15.116-1-pve: 5.15.116-1
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.39-4-pve: 5.15.39-4
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.2
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.0
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.4
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.4
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

/etc/pve/storage.cfg
Code:
dir: local

        path /var/lib/vz

        content vztmpl,iso,backup


lvmthin: local-lvm

        thinpool data

        vgname pve

        content rootdir,images


lvm: drbdlvm

        vgname REDACTED

        content images,rootdir

        shared 1


lvm: backup

        vgname backupvg

        content images,rootdir

        shared 0


qm config 100

Code:
boot: order=scsi0;ide2

cores: 4

ide2: none,media=cdrom

memory: 16384

meta: creation-qemu=7.0.0,ctime=1668077080

name: Ver6.10

net0: virtio=2A:7B:5F:91:FE:F6,bridge=vmbr0,firewall=1

numa: 0

ostype: l26

scsi0: drbdlvm:vm-100-disk-0,size=192G

scsihw: pvscsi

smbios1: uuid=dda29420-a1d8-424e-8688-4e1933cb6bb8

sockets: 1

vmgenid: cc55f130-786d-42d8-9b81-bd0e037c23bc
 
Last edited:
This is one of the moves I tried

Code:
create full clone of drive scsi0 (drbdlvm:vm-100-disk-0)
  Wiping dos signature on /dev/backupvg/vm-100-disk-0.
  Logical volume "vm-100-disk-0" created.
transferred 0.0 B of 192.0 GiB (0.00%)
transferred 1.9 GiB of 192.0 GiB (1.00%)
transferred 3.8 GiB of 192.0 GiB (2.00%)
transferred 5.8 GiB of 192.0 GiB (3.00%)
transferred 7.7 GiB of 192.0 GiB (4.00%)
transferred 9.6 GiB of 192.0 GiB (5.00%)
transferred 11.5 GiB of 192.0 GiB (6.01%)
  Logical volume "vm-100-disk-0" successfully removed.
TASK ERROR: storage migration failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O raw /dev/REDACTED/vm-100-disk-0 /dev/backupvg/vm-100-disk-0' failed: interrupted by signal

I stopped this one after 46 mins giving me a whopping 4.3MB/s
 
Tried again with 8.2.2 but still the same slow transfer.
The trouble seems to be proxmox related because in the Debian shell I get better results:

Code:
root@pve2:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   44660 MB in  2.00 seconds = 22367.20 MB/sec
 Timing buffered disk reads: 734 MB in  3.01 seconds = 244.12 MB/sec
root@pve2:~# dd if=/dev/zero of=/dev/sda bs=8k count=10k
10240+0 records in
10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.361019 s, 232 MB/s
root@pve2:~#

root@pve1:~# hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   46316 MB in  2.00 seconds = 23199.31 MB/sec
 Timing buffered disk reads: 720 MB in  3.01 seconds = 239.38 MB/sec
root@pve1:~# dd if=/dev/zero of=/dev/sda bs=8k count=10k
10240+0 records in
10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.360102 s, 233 MB/s
root@pve1:~#

These results are perfect, but why am I only getting 3 - 4 MB/s when migrating inside Proxmox VE ?
 
Only thing I can think of is some kind of caching issue? The above tests only write 80 MB and the migrate process slows to 4 MB/s after 2 GB of data written:

create full clone of drive scsi1 (local-lvm:vm-100-disk-1)
Logical volume "vm-100-disk-0" created.
transferred 0.0 B of 32.0 GiB (0.00%)
transferred 327.7 MiB of 32.0 GiB (1.00%)
transferred 655.4 MiB of 32.0 GiB (2.00%)
transferred 983.0 MiB of 32.0 GiB (3.00%)
transferred 1.3 GiB of 32.0 GiB (4.00%)
transferred 1.6 GiB of 32.0 GiB (5.00%)
transferred 1.9 GiB of 32.0 GiB (6.01%)
 
Still hoping to start a dialogue here ;)

Doing a dd I get a value of 241 MB/s for 32 GB

root@pve2:~# dd if=/dev/zero of=/dev/sda bs=1536M count=20
20+0 records in
20+0 records out
32212254720 bytes (32 GB, 30 GiB) copied, 133.783 s, 241 MB/s

So why am I getting these terrible values in Proxmox during a disk move ?
 
Moves up to 2GB of data are fast, this one took 22s

create full clone of drive scsi1 (local-lvm:vm-100-disk-1)
Logical volume "vm-100-disk-0" created.
transferred 0.0 B of 2.0 GiB (0.00%)
transferred 21.9 MiB of 2.0 GiB (1.07%)
transferred 44.0 MiB of 2.0 GiB (2.15%)
transferred 65.9 MiB of 2.0 GiB (3.22%)
transferred 88.1 MiB of 2.0 GiB (4.30%)
transferred 110.0 MiB of 2.0 GiB (5.37%)
transferred 132.1 MiB of 2.0 GiB (6.45%)
transferred 154.0 MiB of 2.0 GiB (7.52%)
transferred 175.9 MiB of 2.0 GiB (8.59%)
transferred 198.0 MiB of 2.0 GiB (9.67%)
transferred 220.0 MiB of 2.0 GiB (10.74%)
transferred 242.1 MiB of 2.0 GiB (11.82%)
transferred 264.0 MiB of 2.0 GiB (12.89%)
transferred 285.9 MiB of 2.0 GiB (13.96%)
transferred 308.0 MiB of 2.0 GiB (15.04%)
transferred 329.9 MiB of 2.0 GiB (16.11%)
transferred 352.1 MiB of 2.0 GiB (17.19%)
transferred 374.0 MiB of 2.0 GiB (18.26%)
transferred 396.1 MiB of 2.0 GiB (19.34%)
transferred 418.0 MiB of 2.0 GiB (20.41%)
transferred 439.9 MiB of 2.0 GiB (21.48%)
transferred 462.0 MiB of 2.0 GiB (22.56%)
transferred 483.9 MiB of 2.0 GiB (23.63%)
transferred 506.1 MiB of 2.0 GiB (24.71%)
transferred 528.0 MiB of 2.0 GiB (25.78%)
transferred 550.1 MiB of 2.0 GiB (26.86%)
transferred 572.0 MiB of 2.0 GiB (27.93%)
transferred 593.9 MiB of 2.0 GiB (29.00%)
transferred 616.0 MiB of 2.0 GiB (30.08%)
transferred 638.0 MiB of 2.0 GiB (31.15%)
transferred 660.1 MiB of 2.0 GiB (32.23%)
transferred 682.0 MiB of 2.0 GiB (33.30%)
transferred 704.1 MiB of 2.0 GiB (34.38%)
transferred 726.0 MiB of 2.0 GiB (35.45%)
transferred 747.9 MiB of 2.0 GiB (36.52%)
transferred 770.0 MiB of 2.0 GiB (37.60%)
transferred 792.0 MiB of 2.0 GiB (38.67%)
transferred 814.1 MiB of 2.0 GiB (39.75%)
transferred 836.0 MiB of 2.0 GiB (40.82%)
transferred 857.9 MiB of 2.0 GiB (41.89%)
transferred 880.0 MiB of 2.0 GiB (42.97%)
transferred 901.9 MiB of 2.0 GiB (44.04%)
transferred 924.1 MiB of 2.0 GiB (45.12%)
transferred 946.0 MiB of 2.0 GiB (46.19%)
transferred 968.1 MiB of 2.0 GiB (47.27%)
transferred 990.0 MiB of 2.0 GiB (48.34%)
transferred 1011.9 MiB of 2.0 GiB (49.41%)
transferred 1.0 GiB of 2.0 GiB (50.49%)
transferred 1.0 GiB of 2.0 GiB (51.56%)
transferred 1.1 GiB of 2.0 GiB (52.64%)
transferred 1.1 GiB of 2.0 GiB (53.71%)
transferred 1.1 GiB of 2.0 GiB (54.79%)
transferred 1.1 GiB of 2.0 GiB (55.86%)
transferred 1.1 GiB of 2.0 GiB (56.93%)
transferred 1.2 GiB of 2.0 GiB (58.01%)
transferred 1.2 GiB of 2.0 GiB (59.08%)
transferred 1.2 GiB of 2.0 GiB (60.16%)
transferred 1.2 GiB of 2.0 GiB (61.23%)
transferred 1.2 GiB of 2.0 GiB (62.30%)
transferred 1.3 GiB of 2.0 GiB (63.38%)
transferred 1.3 GiB of 2.0 GiB (64.45%)
transferred 1.3 GiB of 2.0 GiB (65.53%)
transferred 1.3 GiB of 2.0 GiB (66.60%)
transferred 1.4 GiB of 2.0 GiB (67.68%)
transferred 1.4 GiB of 2.0 GiB (68.75%)
transferred 1.4 GiB of 2.0 GiB (69.82%)
transferred 1.4 GiB of 2.0 GiB (70.90%)
transferred 1.4 GiB of 2.0 GiB (71.97%)
transferred 1.5 GiB of 2.0 GiB (73.05%)
transferred 1.5 GiB of 2.0 GiB (74.12%)
transferred 1.5 GiB of 2.0 GiB (75.20%)
transferred 1.5 GiB of 2.0 GiB (76.27%)
transferred 1.5 GiB of 2.0 GiB (77.34%)
transferred 1.6 GiB of 2.0 GiB (78.42%)
transferred 1.6 GiB of 2.0 GiB (79.49%)
transferred 1.6 GiB of 2.0 GiB (80.57%)
transferred 1.6 GiB of 2.0 GiB (81.64%)
transferred 1.7 GiB of 2.0 GiB (82.71%)
transferred 1.7 GiB of 2.0 GiB (83.79%)
transferred 1.7 GiB of 2.0 GiB (84.86%)
transferred 1.7 GiB of 2.0 GiB (85.94%)
transferred 1.7 GiB of 2.0 GiB (87.01%)
transferred 1.8 GiB of 2.0 GiB (88.09%)
transferred 1.8 GiB of 2.0 GiB (89.16%)
transferred 1.8 GiB of 2.0 GiB (90.23%)
transferred 1.8 GiB of 2.0 GiB (91.31%)
transferred 1.8 GiB of 2.0 GiB (92.38%)
transferred 1.9 GiB of 2.0 GiB (93.46%)
transferred 1.9 GiB of 2.0 GiB (94.53%)
transferred 1.9 GiB of 2.0 GiB (95.61%)
transferred 1.9 GiB of 2.0 GiB (96.68%)
transferred 2.0 GiB of 2.0 GiB (97.75%)
transferred 2.0 GiB of 2.0 GiB (98.83%)
transferred 2.0 GiB of 2.0 GiB (99.90%)
transferred 2.0 GiB of 2.0 GiB (100.00%)
transferred 2.0 GiB of 2.0 GiB (100.00%)
Logical volume "vm-100-disk-1" successfully removed.
TASK OK
 
just 0.2 GB extra took 75 seconds more

this is the move of a 2.2 GB disk in 1m37s

create full clone of drive scsi1 (local-lvm:vm-100-disk-1)
Logical volume "vm-100-disk-0" created.
transferred 0.0 B of 2.2 GiB (0.00%)
transferred 23.9 MiB of 2.2 GiB (1.06%)
transferred 48.1 MiB of 2.2 GiB (2.13%)
transferred 72.0 MiB of 2.2 GiB (3.19%)
transferred 96.1 MiB of 2.2 GiB (4.26%)
transferred 120.0 MiB of 2.2 GiB (5.32%)
transferred 143.9 MiB of 2.2 GiB (6.38%)
transferred 168.1 MiB of 2.2 GiB (7.45%)
transferred 192.0 MiB of 2.2 GiB (8.51%)
transferred 215.9 MiB of 2.2 GiB (9.57%)
transferred 240.0 MiB of 2.2 GiB (10.64%)
transferred 264.0 MiB of 2.2 GiB (11.70%)
transferred 288.1 MiB of 2.2 GiB (12.77%)
transferred 312.0 MiB of 2.2 GiB (13.83%)
transferred 335.9 MiB of 2.2 GiB (14.89%)
transferred 360.1 MiB of 2.2 GiB (15.96%)
transferred 384.0 MiB of 2.2 GiB (17.02%)
transferred 408.1 MiB of 2.2 GiB (18.09%)
transferred 432.0 MiB of 2.2 GiB (19.15%)
transferred 455.9 MiB of 2.2 GiB (20.21%)
transferred 480.1 MiB of 2.2 GiB (21.28%)
transferred 504.0 MiB of 2.2 GiB (22.34%)
transferred 527.9 MiB of 2.2 GiB (23.40%)
transferred 552.0 MiB of 2.2 GiB (24.47%)
transferred 576.0 MiB of 2.2 GiB (25.53%)
transferred 600.1 MiB of 2.2 GiB (26.60%)
transferred 624.0 MiB of 2.2 GiB (27.66%)
transferred 647.9 MiB of 2.2 GiB (28.72%)
transferred 672.1 MiB of 2.2 GiB (29.79%)
transferred 696.0 MiB of 2.2 GiB (30.85%)
transferred 719.9 MiB of 2.2 GiB (31.91%)
transferred 744.0 MiB of 2.2 GiB (32.98%)
transferred 767.9 MiB of 2.2 GiB (34.04%)
transferred 792.1 MiB of 2.2 GiB (35.11%)
transferred 816.0 MiB of 2.2 GiB (36.17%)
transferred 839.9 MiB of 2.2 GiB (37.23%)
transferred 864.0 MiB of 2.2 GiB (38.30%)
transferred 888.0 MiB of 2.2 GiB (39.36%)
transferred 912.1 MiB of 2.2 GiB (40.43%)
transferred 936.0 MiB of 2.2 GiB (41.49%)
transferred 959.9 MiB of 2.2 GiB (42.55%)
transferred 984.1 MiB of 2.2 GiB (43.62%)
transferred 1008.0 MiB of 2.2 GiB (44.68%)
transferred 1.0 GiB of 2.2 GiB (45.74%)
transferred 1.0 GiB of 2.2 GiB (46.81%)
transferred 1.1 GiB of 2.2 GiB (47.87%)
transferred 1.1 GiB of 2.2 GiB (48.94%)
transferred 1.1 GiB of 2.2 GiB (50.00%)
transferred 1.1 GiB of 2.2 GiB (51.06%)
transferred 1.1 GiB of 2.2 GiB (52.13%)
transferred 1.2 GiB of 2.2 GiB (53.19%)
transferred 1.2 GiB of 2.2 GiB (54.26%)
transferred 1.2 GiB of 2.2 GiB (55.32%)
transferred 1.2 GiB of 2.2 GiB (56.38%)
transferred 1.3 GiB of 2.2 GiB (57.45%)
transferred 1.3 GiB of 2.2 GiB (58.51%)
transferred 1.3 GiB of 2.2 GiB (59.57%)
transferred 1.3 GiB of 2.2 GiB (60.64%)
transferred 1.4 GiB of 2.2 GiB (61.70%)
transferred 1.4 GiB of 2.2 GiB (62.77%)
transferred 1.4 GiB of 2.2 GiB (63.83%)
transferred 1.4 GiB of 2.2 GiB (64.89%)
transferred 1.5 GiB of 2.2 GiB (65.96%)
transferred 1.5 GiB of 2.2 GiB (67.02%)
transferred 1.5 GiB of 2.2 GiB (68.09%)
transferred 1.5 GiB of 2.2 GiB (69.15%)
transferred 1.5 GiB of 2.2 GiB (70.21%)
transferred 1.6 GiB of 2.2 GiB (71.28%)
transferred 1.6 GiB of 2.2 GiB (72.34%)
transferred 1.6 GiB of 2.2 GiB (73.40%)
transferred 1.6 GiB of 2.2 GiB (74.47%)
transferred 1.7 GiB of 2.2 GiB (75.53%)
transferred 1.7 GiB of 2.2 GiB (76.60%)
transferred 1.7 GiB of 2.2 GiB (77.66%)
transferred 1.7 GiB of 2.2 GiB (78.72%)
transferred 1.8 GiB of 2.2 GiB (79.79%)
transferred 1.8 GiB of 2.2 GiB (80.85%)
transferred 1.8 GiB of 2.2 GiB (81.91%)
transferred 1.8 GiB of 2.2 GiB (82.98%)
transferred 1.9 GiB of 2.2 GiB (84.04%)
transferred 1.9 GiB of 2.2 GiB (85.11%)
transferred 1.9 GiB of 2.2 GiB (86.17%)
transferred 1.9 GiB of 2.2 GiB (87.23%)
transferred 1.9 GiB of 2.2 GiB (88.30%)
transferred 2.0 GiB of 2.2 GiB (89.36%)
transferred 2.0 GiB of 2.2 GiB (90.43%)
transferred 2.0 GiB of 2.2 GiB (91.49%)
transferred 2.0 GiB of 2.2 GiB (92.55%)
transferred 2.1 GiB of 2.2 GiB (93.62%)
transferred 2.1 GiB of 2.2 GiB (94.68%)
transferred 2.1 GiB of 2.2 GiB (95.74%)
transferred 2.1 GiB of 2.2 GiB (96.81%)
transferred 2.2 GiB of 2.2 GiB (97.87%)
transferred 2.2 GiB of 2.2 GiB (98.94%)
transferred 2.2 GiB of 2.2 GiB (100.00%)
transferred 2.2 GiB of 2.2 GiB (100.00%)
transferred 2.2 GiB of 2.2 GiB (100.00%)
Logical volume "vm-100-disk-1" successfully removed.
TASK OK
 
Last edited:
The question: why does proxmox almost grind to a halt after copying 2.0 GB of data ?
 
This happened in some version of Proxmox because I used to have normal transfer speeds in Proxmox 7.x or 6.x (it's been a while since I used the extra disk)
 
Last edited:
Am I really the only person here with this issue? I cannot believe this.

I have this issue on 4 different systems (2 asus and 2 supermicro). The only thing they have in common is the same harddisk type and RAID controller type.
 
Some other test: copied a 2.2 GB file of random data to that single disk and that only took a couple of seconds.

I don't believe this is a hardware issue with all the tests done.

This is some Proxmox bug
 
To what storage type are you moving the disks to? Because dd'ing zeros to it is usually not a good benchmark. On ZFS that would result in almost no data written to the disks at all.

What kind of disks are in use? Are they consumer SSDs or datacenter ones with PLP?

Are there other tasks / guests writing data to the same disk at that time?
 
The storage is a HUS726T4TALE6L4 WD Ultrastar DC connected as JBOD on a LSI MR9341-4i

I have tried several sources;

1. RAID1 with the same disks as the one above
2. Samsung MZ7L3240HCHQ-00A07

There is no speed difference between the sources.

And only the write to the JBOD is slow.

Writes to the RAID and the SSD are fast. Also from the JBOD

no other tasks running, I have tried with all guests off.

And some extra info: I have another system with the same disk setup but with the LSI MR 9361-8i
That system has fast moves to the JBOD.

I will now upgrade my subscription from community to standard to get things going
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!