VM migration speed question

P

Pavletto

New Member
Sep 13, 2023
12
1
3
43
Hi collegues,
i would like to ask you about migration speed between PVE cluster nodes.
I have a 3-node PVE 8 cluster with 2x40G network links: one for CEPH cluster (1) and another one for PVE cluster/CEPH public network (2).
CEPH OSDs is all-nvme.
In cluster options i've set also one of these 40G netwoks for migration (2)

When i migrating VM with 32G RAM from one node to another i get these results:
Code: 
2023-09-18 11:01:18 starting migration of VM 100 to node 'pve-up-1' (10.100.41.30)
2023-09-18 11:01:18 starting VM 100 on remote node 'pve-up-1'
2023-09-18 11:01:20 start remote tunnel
2023-09-18 11:01:20 ssh tunnel ver 1
2023-09-18 11:01:20 starting online/live migration on unix:/run/qemu-server/100.migrate
2023-09-18 11:01:20 set migration capabilities
2023-09-18 11:01:20 migration downtime limit: 100 ms
2023-09-18 11:01:20 migration cachesize: 4.0 GiB
...
2023-09-18 11:02:20 xbzrle: send updates to 65487 pages in 7.9 MiB encoded memory, cache-miss 84.62%, overflow 604
2023-09-18 11:02:21 auto-increased downtime to continue migration: 200 ms
2023-09-18 11:02:22 migration active, transferred 28.6 GiB of 32.1 GiB VM-state, 287.9 MiB/s
2023-09-18 11:02:22 xbzrle: send updates to 190269 pages in 50.6 MiB encoded memory, cache-miss 34.56%, overflow 861
2023-09-18 11:02:24 migration active, transferred 28.6 GiB of 32.1 GiB VM-state, 151.9 MiB/s, VM dirties lots of memory: 299.1 MiB/s
2023-09-18 11:02:24 xbzrle: send updates to 306352 pages in 60.5 MiB encoded memory, cache-miss 27.66%, overflow 1180
2023-09-18 11:02:25 auto-increased downtime to continue migration: 400 ms
2023-09-18 11:02:27 average migration speed: 491.1 MiB/s - downtime 531 ms
2023-09-18 11:02:27 migration status: completed
2023-09-18 11:02:28 Waiting for spice server migration
2023-09-18 11:02:30 migration finished successfully (duration 00:01:13)

After some googling i've set "insecure" type for migration:

Code: 
/etc/pve/datacenter.cfg

migration: network=10.100.41.10/24,type=insecure

After this change result is:

Code: 
2023-09-18 11:18:03 use dedicated network address for sending migration traffic (10.100.41.20)
2023-09-18 11:18:03 starting migration of VM 100 to node 'pve-down-2' (10.100.41.20)
2023-09-18 11:18:03 starting VM 100 on remote node 'pve-down-2'
2023-09-18 11:18:05 start remote tunnel
2023-09-18 11:18:06 ssh tunnel ver 1
2023-09-18 11:18:06 starting online/live migration on tcp:10.100.41.20:60000
2023-09-18 11:18:06 set migration capabilities
2023-09-18 11:18:06 migration downtime limit: 100 ms
2023-09-18 11:18:06 migration cachesize: 4.0 GiB
...
2023-09-18 11:18:37 xbzrle: send updates to 47084 pages in 4.1 MiB encoded memory, cache-miss 78.66%, overflow 294
2023-09-18 11:18:38 auto-increased downtime to continue migration: 200 ms
2023-09-18 11:18:39 migration active, transferred 28.2 GiB of 32.1 GiB VM-state, 179.1 MiB/s
2023-09-18 11:18:39 xbzrle: send updates to 129779 pages in 20.8 MiB encoded memory, cache-miss 16.74%, overflow 357
2023-09-18 11:18:40 auto-increased downtime to continue migration: 400 ms
2023-09-18 11:18:41 migration active, transferred 28.2 GiB of 32.1 GiB VM-state, 104.6 MiB/s, VM dirties lots of memory: 177.4 MiB/s
2023-09-18 11:18:41 xbzrle: send updates to 220757 pages in 42.1 MiB encoded memory, cache-miss 20.16%, overflow 424
2023-09-18 11:18:42 auto-increased downtime to continue migration: 800 ms
2023-09-18 11:18:44 migration active, transferred 28.3 GiB of 32.1 GiB VM-state, 339.8 MiB/s
2023-09-18 11:18:44 xbzrle: send updates to 339327 pages in 50.7 MiB encoded memory, cache-miss 21.37%, overflow 620
2023-09-18 11:18:46 average migration speed: 822.5 MiB/s - downtime 472 ms
2023-09-18 11:18:46 migration status: completed
2023-09-18 11:18:47 Waiting for spice server migration
2023-09-18 11:18:49 migration finished successfully (duration 00:00:47)

Speed is increased but whis is a way too far from capabilities of 40G network and disk speed:

Code: 
root@pve-down-1:/tmp# rados bench -p scbench 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_pve-down-1_1588020
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       615       599   2395.85      2396   0.0213653   0.0262351
    2      16      1238      1222    2443.8      2492    0.164208   0.0239879
    3      16      1605      1589   2118.48      1468   0.0379306   0.0300604
    4      16      2261      2245    2244.8      2624   0.0292071   0.0283965
    5      16      2949      2933   2346.19      2752    0.015009   0.0272138
    6      16      3687      3671   2447.11      2952   0.0131768   0.0261067
    7      16      4417      4401   2514.63      2920   0.0262535     0.02521
    8      16      5089      5073   2536.27      2688    0.010768   0.0251883
    9      16      5756      5740   2550.88      2668   0.0152786   0.0250424
   10      15      6397      6382   2552.56      2568  0.00827939   0.0247853
   11       6      6397      6391   2323.79        36    0.218062   0.0248901
   12       6      6397      6391   2130.14         0           -   0.0248901
Total time run:         12.5638
Total writes made:      6397
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     2036.65
Stddev Bandwidth:       1056.8
Max bandwidth (MB/sec): 2952
Min bandwidth (MB/sec): 0
Average IOPS:           509
Stddev IOPS:            264.2
Max IOPS:               738
Min IOPS:               0
Average Latency(s):     0.0274665
Stddev Latency(s):      0.0937761
Max latency(s):         3.00742
Min latency(s):         0.00576705

May be there is more parameters i should use too speed up a migration process?
 
Hi,

Thank you for sharing the outputs!

Pavletto said:
After some googling i've set "insecure" type for migration:
Click to expand...
This is described in `man datacenter.cfg`.

May I ask you about the MTU in the 10.100.41.10/24 network? If it's 1500 consider configuring it to use frames, e.g., 9000, and see if that will help.
 
  • Like
Reactions: Pavletto
Thanks for your reply.
I've set MTU 9000 on interface, then in vmbr, and finally on vlan if

Code: 
auto ens6f1
iface ens6f1 inet static
        address 10.100.40.20/24
#40G iface CEPH

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.160/23
        gateway 192.168.10.145
        bridge-ports ens9f0np0
        bridge-stp off
        bridge-fd 0
#10G MGMT

auto vmbr10
iface vmbr10 inet manual
        bridge-ports ens9f1np1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10G Trunk NW

auto vmbr40
iface vmbr40 inet manual
        bridge-ports ens6f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
#40G Trunk NW

auto vlan41
iface vlan41 inet static
        address 10.100.41.20/24
        mtu 9000
        vlan-raw-device vmbr40
#40G VLAN41 CLUSTER_NW

auto vlan9
iface vlan9 inet static
        address 192.168.25.20/24
        vlan-raw-device vmbr40
#40G VLAN9 192.168.25.20

Here is results:
Code: 
2023-09-18 13:09:37 starting migration of VM 100 to node 'pve-up-1' (10.100.41.30)
2023-09-18 13:09:37 starting VM 100 on remote node 'pve-up-1'
2023-09-18 13:09:39 start remote tunnel
2023-09-18 13:09:40 ssh tunnel ver 1
2023-09-18 13:09:40 starting online/live migration on tcp:10.100.41.30:60000
2023-09-18 13:09:40 set migration capabilities
2023-09-18 13:09:40 migration downtime limit: 100 ms
2023-09-18 13:09:40 migration cachesize: 4.0 GiB
...
2023-09-18 13:10:00 auto-increased downtime to continue migration: 200 ms
2023-09-18 13:10:01 migration active, transferred 28.2 GiB of 32.1 GiB VM-state, 397.5 MiB/s
2023-09-18 13:10:01 xbzrle: send updates to 92212 pages in 6.2 MiB encoded memory, cache-miss 26.87%, overflow 208
2023-09-18 13:10:02 auto-increased downtime to continue migration: 400 ms
2023-09-18 13:10:03 migration active, transferred 28.3 GiB of 32.1 GiB VM-state, 156.4 MiB/s, VM dirties lots of memory: 190.0 MiB/s
2023-09-18 13:10:03 xbzrle: send updates to 188758 pages in 13.6 MiB encoded memory, cache-miss 20.51%, overflow 354
2023-09-18 13:10:04 average migration speed: 1.3 GiB/s - downtime 457 ms
2023-09-18 13:10:04 migration status: completed
2023-09-18 13:10:06 Waiting for spice server migration
2023-09-18 13:10:07 migration finished successfully (duration 00:00:30)

So more progress achieved but always want some more-)
 
vm memory transfert can reach around 10gbit/s (seem that you already have 1,3GiB/s , so around 10gbit/s).
Click to expand...
So maximum VM migration speed is limited to approx. 10Gbit/s? Did i understood correctly?
 
spirit said:
vm memory transfert can reach around 10gbit/s (seem that you already have 1,3GiB/s , so around 10gbit/s).
Click to expand...
Just wanted to chime in—I’m seeing a bit higher speeds in my setup. I’m using MCX314A-BCCT NICs (56 Gb/s) on a dedicated network for migration traffic, and I typically hit an average of around 3.1 GiB/s (~26.6 Gbit/s) during migrations.

It seems like with the right hardware, you can definitely get higher transfer rates than 10 Gbit/s. Thought I’d share my experience!

Cheers!
 
Last edited:
Pavletto said:
So maximum VM migration speed is limited to approx. 10Gbit/s? Did i understood correctly?
Click to expand...

Sorry to dig up this semi-old thread but I thought I'd also chime in.

I have a small cluster I'm working on and I've been able to achieve 2.1GiB/s this morning.
Server: Dell R240, 64GB memory, ZFS Raid 10 (4x960GB SSD)
Switch: Mellanox SX1024
NIC: 2xPort Mellanox Connect (MT27500)
Networking currently set at 40Gbps-Ethernet
MTU = 9000 on NIC and Switch Ports
Migration Network = insecure.

I was testing with a VM size of 48GB,

2024-12-16 12:06:13 set migration capabilities
2024-12-16 12:06:13 migration downtime limit: 100 ms
2024-12-16 12:06:13 migration cachesize: 512.0 MiB
2024-12-16 12:06:13 set migration parameters
2024-12-16 12:06:13 start migrate command to tcp:192.168.161.10:60000
2024-12-16 12:06:14 migration active, transferred 2.7 GiB of 4.1 GiB VM-state, 2.4 GiB/s
2024-12-16 12:06:15 average migration speed: 2.1 GiB/s - downtime 38 ms
2024-12-16 12:06:15 migration status: completed
all 'mirror' jobs are ready
drive-scsi0: Completing block job...
drive-scsi0: Completed successfully.
drive-scsi0: mirror-job finished
2024-12-16 12:06:17 stopping NBD storage migration server on target.
2024-12-16 12:06:22 migration finished successfully (duration 00:01:25)
TASK OK


Still working on trying to see if I can further improve the performance.
 
You must log in or register to reply here.
Top Bottom