Hi all,
I'm migrating VMs from a Ceph backed 7.4 cluster (all enterprise SSD, 40Gbit dual network etc, no problems there), to a new standalone 8.4 node (RAID 10 enterprise SSD on H740p etc) using LVM thin local storage. Also connected via the dual 40g.
Normally the new 8.4 node is great, 0% iowait, load under 3 etc, performance is fantastic as expected. However, when using PDM to migrate a VM from the Ceph based 7.4 cluster, for the initial part, before it even gets to the percentage based transfer of the virtual disk, if it's 1TB for example, it'll take about an hour of high intensity writes whilst the transfer sits on this:
At this point, new node has high IOwait, there's no network activity, and iotop shows several hundred MB/sec of writes, but not against any actual process. It's as if proxmox/qemu is pre-writing something, but not sure what. Load goes up to about 20, and stays there.
Then as soon as that finishes, you get the:
and so on, until it finishes. During this phase, iowait goes back to normal, load drops back to normal 2-3, network activity is high as the actual virtual disk data is transferred.
What's it actually doing in the first part that causes the high IOwait and load? It's not actually transferring anything, is it just writing a blank virtual disk in the thin-lvm or something? How can we mitigate this please?
I'm migrating VMs from a Ceph backed 7.4 cluster (all enterprise SSD, 40Gbit dual network etc, no problems there), to a new standalone 8.4 node (RAID 10 enterprise SSD on H740p etc) using LVM thin local storage. Also connected via the dual 40g.
Normally the new 8.4 node is great, 0% iowait, load under 3 etc, performance is fantastic as expected. However, when using PDM to migrate a VM from the Ceph based 7.4 cluster, for the initial part, before it even gets to the percentage based transfer of the virtual disk, if it's 1TB for example, it'll take about an hour of high intensity writes whilst the transfer sits on this:
Code:
2025-08-20 20:53:13 scsi0: start migration to nbd:unix:/run/qemu-server/112_nbd.migrate:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
tunnel: accepted new connection on '/run/qemu-server/112_nbd.migrate'
tunnel: requesting WS ticket via tunnel
tunnel: established new WS for forwarding '/run/qemu-server/112_nbd.migrate'At this point, new node has high IOwait, there's no network activity, and iotop shows several hundred MB/sec of writes, but not against any actual process. It's as if proxmox/qemu is pre-writing something, but not sure what. Load goes up to about 20, and stays there.
Then as soon as that finishes, you get the:
Code:
drive-scsi0: transferred 0.0 B of 1.0 TiB (0.00%) in 52m 22s
drive-scsi0: transferred 129.0 MiB of 1.0 TiB (0.01%) in 52m 23s
drive-scsi0: transferred 241.0 MiB of 1.0 TiB (0.02%) in 52m 24s
drive-scsi0: transferred 369.0 MiB of 1.0 TiB (0.04%) in 52m 25s
drive-scsi0: transferred 497.0 MiB of 1.0 TiB (0.05%) in 52m 26s
drive-scsi0: transferred 625.0 MiB of 1.0 TiB (0.06%) in 52m 27s
drive-scsi0: transferred 788.0 MiB of 1.0 TiB (0.08%) in 52m 28s
drive-scsi0: transferred 912.0 MiB of 1.0 TiB (0.09%) in 52m 29s
drive-scsi0: transferred 1.0 GiB of 1.0 TiB (0.10%) in 52m 30s
drive-scsi0: transferred 1.2 GiB of 1.0 TiB (0.11%) in 52m 31s
drive-scsi0: transferred 1.3 GiB of 1.0 TiB (0.13%) in 52m 32s
drive-scsi0: transferred 1.4 GiB of 1.0 TiB (0.14%) in 52m 33sand so on, until it finishes. During this phase, iowait goes back to normal, load drops back to normal 2-3, network activity is high as the actual virtual disk data is transferred.
What's it actually doing in the first part that causes the high IOwait and load? It's not actually transferring anything, is it just writing a blank virtual disk in the thin-lvm or something? How can we mitigate this please?