Slow Migration between nodes

[drjaymz@]

I recently set up three nodes running a fairly small VM which consists of 3 disks 25/4/25 Gb so 54Gb in total. Migration of that VM from one node to another tops out at 120 Megabit/s. You can quite clearly see the limitation on the IO graphs.
I can quite happily get 950 Megabits between nodes for other data or doing a backup to a network drive so why is this bit so slow? There isn't much in the way of contention but the I'd expect migration to be < 10 mins not over an hour.
The IO limitation is so flat that it looks exactly like bandwidth limitation is set.

I think what's its doing is creating a snapshot, then compressing that (which goes to about 10Gb) and then pushing that over - but its terrible performance.

I do have replication for the nodes and that only takes a minute or two every 15 mins with the changes we accumulate - but the initial sync, I think went similarly slow.

The nodes do have additional 10GBit ports which I plan to use for replication, but the replication isn't using more than 10% of the primary to start with.

I have looked around and found others notes slow behaviour after the nodes have been running a long time, but these are only a week old. So I don't know what else to check.

 

[B.Otto]

The default migration method is secure, so via SSH iirc. If the migration network is local, you could try to use the insecure mode by editing the /etc/pve/datacenter.cfg and setting migration: insecure

Is the migration from local to local storage or are you using a shared storage like ceph?

In the case of local-to-local the low performance could also be storage-related, since it has to read all data (somewhat fast), send it to the second node (usually fast) and write it to the new storage (usually slowest).

In the case of a shared storage, a migration only needs to copy the RAM of the machine (plus delta-blocks that changed while the migration was running), so it is much faster.

 

[drjaymz@]

its local storage, but I can write to that local storage an order of magnitude quicker unless something is horribly misconfigured somewhere.
To rule that out - I could try a backup restore to another node to see what speed that writes at - that should be coming in from the network at close to full bandwidth and then decompressing and writing to the disk at a reasonable rate. If its also strangled then definitely something wonky there.

Shared storage may be an answer and something I need to look at right now whilst I only have one VM on there.

 

[drjaymz@]

It restored in about 11 minutes 54Gb. That means that the limitation was NOT related to write performance of the disk, in this case it was read from the network performance. So, that means its unsolved as of now.

I will try the unsecure copy and see if that makes any difference.

 

[drjaymz@]

It restored in about 11 minutes 54Gb. That means that the limitation was NOT related to write performance of the disk, in this case it was read from the network performance. So, that means its unsolved as of now.

I will try the unsecure copy and see if that makes any difference.

OK that is copying very quickly now.



Sorry there is such a long gap in the middle but you can see the inbound copy was 12MByte/s The outbound is now 120MBytes. So yeah - its the secure method that slows it down by exactly 90.0%.

 

[drjaymz@]

scratch that - its related to that first node. Its exactly the same migrating back. 10% speed.

Yep, pushing it back the way....



You can see its now 10% - that looks like its deliberately capped somewhere?