"Slow" VM Migration Speeds

[wbarnard81]

Hi All.

It feels like I am going to complain about something with a golden spoon in my mouth, but this is our production servers...

When I migrate a VM form one node to another, in the cluster, it seems like it tops out at 800Mbps-ish. Some information first...

Hardware I am using:
AMD EPYC 7702P 64-Core Processor
512GB DDR4 3200Mhz
2x Micron 7300 480GB (Raid1 zfs for boot)
6x Kioxia KCD6XLUL960G 960GB NVMe
Mellanox ConnectX 4 (25Gbps) NICs, connecting with DAC Cables.

Disk setup for the 6x NVMe:
zpool create -f -o ashift=12 houmyvas mirror /dev/nvme0n1 /dev/nvme1n1 mirror /dev/nvme2n1 /dev/nvme3n1 mirror /dev/nvme6n1 /dev/nvme7n1

Source node is using Proxmox 7.2 and Kernel 5.15 and Destination node is Proxmox 7.3 and Kernel 6.1.





Yes, the screen shot above shows speed when migrating the memory, but the one above that should "show" that it is the same when migrating the HDD.

So my questions:
1. How can I figure out what would be the bottleneck here?
2. What happens to the data that is written to the HDD, while the migration takes place?

 

[muaboka]

how did you achieve these speeds? is it an --online migration or offline? just out of curiosity. I have a cluster each with 1Gig for corosync with 2 interfaces on each node connected as redundant (active-backup). However I am achieving migration speeds of around 100MiB max. What am I doing wrong here?

 

[BobhWasatch]

Um, 1 gigabit = 1000 megabits = 1000/8 megabytes = 125 megabytes. With overhead and required dead times between frames 100 MiB is about what you would expect on a gigabit LAN.

ETA: The OP has a 25 gigabit LAN.

 

[UdoB]

how did you achieve these speeds?

I am not @wbarnard81 but he wrote: "Mellanox ConnectX 4 (25Gbps) NICs, connecting with DAC Cables." - and with these 800 MByte/s is... slow

 

[muaboka]

Um, 1 gigabit = 1000 megabits = 1000/8 megabytes = 125 megabytes. With overhead and required dead times between frames 100 MiB is about what you would expect on a gigabit LAN.

ETA: The OP has a 25 gigabit LAN.

Thanks for the math. I will get to buying some 25G cards then.

 

[wbarnard81]

I am not @wbarnard81 but he wrote: "Mellanox ConnectX 4 (25Gbps) NICs, connecting with DAC Cables." - and with these 800 MByte/s is... slow

Yeah, I still don't know where my "bottleneck" is. :-(

 

[fabian]

please post the full migration task log and "pveversion -v"