[SOLVED] Errors/warnings and very slow migration between nodes for VM

[Gerk]

4 node cluster. Bonded (LACP) 10Gb NIC backend (2 interfaces per node). Using Ceph storage (6 x nvme 2TB, 4 OSDs per nvme). All other disk operations are quite fast. VM disk is stored on ceph.

The state transfer is going soooooo slooooow on a test migration. Not sure if this is expected or something I've mucked up, but seeing all the warnings around the Perl module for migration makes me concerned. Running latest updates for 8.4.1 as of this morning. Just ramping up to put this cluster into production but I need to resolve this super slow speed. I can't imagine HA would even work properly with this speed!

2025-05-22 11:27:39 migration active, transferred 27.7 MiB of 4.0 GiB VM-state, 244.0 KiB/s
Use of uninitialized value in string ne at /usr/share/perl5/PVE/QemuMigrate.pm line 1324.
Use of uninitialized value $last_vfio_transferred in string ne at /usr/share/perl5/PVE/QemuMigrate.pm line 1324.

 

[leesteken]

Looks like this is resolved in the other thread: https://forum.proxmox.com/threads/e...tion-between-nodes-with-vm.166551/post-772873

 

[Gerk]

Yes somehow I think this double posted. This forum software is a bit odd LOL.

 

[waltar]

Actual pve 8.4.1 5-node cluster with 10Gb network and all pve images on nfs server, here enable maintenance mode on dell T620, it things about a couple of seconds before to begin but then auto-migration start of ha defined 8 vm (+1 lxc) and take 10s for all combined (all to only 1 other node (as we have couple different hw nodes) with other vm load) as seen with parallel running a "sar -n DEV 1 --iface=enp1s0,eno1,eno2" :
...
05:20:27 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:28 PM eno1 147.00 190.00 27.83 67.86 0.00 0.00 1.00 0.06
05:20:28 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:28 PM enp1s0 96.00 113.00 20.52 57.71 0.00 0.00 0.00 0.00

05:20:28 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:29 PM eno1 79.00 85.00 15.70 15.20 0.00 0.00 0.00 0.01
05:20:29 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:29 PM enp1s0 14971.00 201955.00 968.73 298523.33 0.00 0.00 0.00 24.46 ##### start #####

05:20:29 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:30 PM eno1 132.00 123.00 30.52 21.39 0.00 0.00 1.00 0.03
05:20:30 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:30 PM enp1s0 57172.00 812678.00 3692.75 1201543.80 0.00 0.00 0.00 98.43

05:20:30 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:31 PM eno1 67.00 79.00 12.32 14.40 0.00 0.00 0.00 0.01
05:20:31 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:31 PM enp1s0 40276.00 812733.00 2602.11 1201591.63 0.00 0.00 0.00 98.43

05:20:31 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:32 PM eno1 242.00 239.00 68.43 58.09 0.00 0.00 1.00 0.06
05:20:32 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:32 PM enp1s0 50884.00 813080.00 3289.18 1202013.93 0.00 0.00 0.00 98.47

05:20:32 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:33 PM eno1 120.00 177.00 21.24 45.26 0.00 0.00 0.00 0.04
05:20:33 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:33 PM enp1s0 39846.00 812583.00 2581.19 1201295.99 0.00 0.00 0.00 98.41

05:20:33 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:34 PM eno1 395.00 390.00 105.23 78.73 0.00 0.00 1.00 0.09
05:20:34 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:34 PM enp1s0 34389.00 809254.00 2283.96 1196415.64 0.00 0.00 0.00 98.01

05:20:34 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:35 PM eno1 276.00 330.00 55.19 111.11 0.00 0.00 0.00 0.09
05:20:35 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:35 PM enp1s0 28346.00 792801.00 1864.44 1172089.74 0.00 0.00 0.00 96.02

05:20:35 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:36 PM eno1 279.00 335.00 55.57 88.67 0.00 0.00 1.00 0.07
05:20:36 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:36 PM enp1s0 24137.00 763710.00 1613.74 1128941.44 0.00 0.00 0.00 92.48

05:20:36 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:37 PM eno1 293.00 341.00 52.09 96.77 0.00 0.00 6.00 0.08
05:20:37 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:37 PM enp1s0 14917.00 581874.00 983.39 860198.25 0.00 0.00 0.00 70.47

05:20:37 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:38 PM eno1 250.00 290.00 42.23 103.26 0.00 0.00 18.00 0.08
05:20:38 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:38 PM enp1s0 5544.00 209964.00 366.09 310301.20 0.00 0.00 0.00 25.42 ##### done #####

05:20:38 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:20:39 PM eno1 206.00 244.00 43.91 103.13 0.00 0.00 2.00 0.08
05:20:39 PM eno2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:20:39 PM enp1s0 42.00 75.00 8.77 62.60 0.00 0.00 0.00 0.01
...