Disk migration speed is capped by disk read/write limits in PVE 9

[jnkraft]

I recently upgraded my cluster from PVE8 to PVE9 and noticed that offline and online disk migration (qcow2) between storage backends (NFS) has become SLOW. After a few experiments, I observed that the speed is constant and, surprisingly, matches the Read and Write limits set in the settings of the disk being migrated. On PVE8, on my 10Gb network, I had a decent migration speed (close to saturating the link), but now it was 100 MB/s, which exactly matched the limits configured for that disk. After removing the limits, the speed became what it should be. If I set limits to 30 MB/s, I get a migration speed of 30 MB/s. I didn’t check the impact of the IOPS limits or the burst limits for both limit types.

Since I have a lot of VMs in the cluster (it’s a work cluster), it’s a highly competitive virtualization environment in terms of IOPS and bandwidth, so most VMs have some kind of limits configured.

I can say with complete certainty that neither PVE7 nor PVE8 showed this behavior. A very unexpected new "feature". What did I miss in the changelog?

 

[bbgeek17]

Hi @jnkraft , this thread may be somewhat helpful:

M

Thread 'bandwidth limits'

Feb 9, 2025

bwlimit: [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>] [,move=<LIMIT>] [,restore=<LIMIT>]
Set I/O bandwidth limit for various operations (in KiB/s).

clone=<LIMIT>
bandwidth limit in KiB/s for cloning disks

default=<LIMIT>
default bandwidth limit in KiB/s

migration=<LIMIT>
bandwidth limit in KiB/s for migrating guests (including moving local disks)

move=<LIMIT>
bandwidth limit in KiB/s for moving disks

restore=<LIMIT>
bandwidth limit in KiB/s for restoring guests from backups

Which of these limits applies for migration jobs?
Are the limits "per job" or "collectively for all jobs...

• michabbs
• Replies: 3
• Forum: Proxmox VE: Installation and configuration

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jnkraft]

Hi @jnkraft , this thread may be somewhat helpful:

M

Thread 'bandwidth limits'

Feb 9, 2025

bwlimit: [clone=<LIMIT>] [,default=<LIMIT>] [,migration=<LIMIT>] [,move=<LIMIT>] [,restore=<LIMIT>]
Set I/O bandwidth limit for various operations (in KiB/s).

clone=<LIMIT>
bandwidth limit in KiB/s for cloning disks

default=<LIMIT>
default bandwidth limit in KiB/s

migration=<LIMIT>
bandwidth limit in KiB/s for migrating guests (including moving local disks)

move=<LIMIT>
bandwidth limit in KiB/s for moving disks

restore=<LIMIT>
bandwidth limit in KiB/s for restoring guests from backups

Which of these limits applies for migration jobs?
Are the limits "per job" or "collectively for all jobs...

• michabbs
• Replies: 3
• Forum: Proxmox VE: Installation and configuration

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi, thanks for the link — I read through that thread. My limits are set to default/none, and, as I mentioned, I have a reproducible correlation between migration speed and the disk limits. And I upgraded from PVE 8 to PVE 9 last weekend, so the contrast is stark and very recent.

 

[fiona]

Hi,
thank you for the report! I can reproduce the issue here and will look into it. It's a regression from the switch to the QEMU -blockdev command line with machine versions >= 10.0. We accidentally use the throttled block node as the source of the mirror. Mirroring in QEMU currently only allows using top-level nodes as the source and those include the throttling, so fixing this will require patching QEMU itself too.

 

[jnkraft]

Hi,
thank you for the report! I can reproduce the issue here and will look into it. It's a regression from the switch to the QEMU -blockdev command line with machine versions >= 10.0. We accidentally use the throttled block node as the source of the mirror. Mirroring in QEMU currently only allows using top-level nodes as the source and those include the throttling, so fixing this will require patching QEMU itself too.

Hello!
Thanks for confirming the bug. Will wait for a fix.