qm remote-migrate slow over WAN

[Solinus]

Hi there folks.

I have a datacenter I'm trying to vacate and am attempting to use the qm remote-migrate process to migrate the new cluster. I have been able to successfully start a migration but it is dog slow despite the fact that I have 1 gig symmetrical fiber at the old location and 8 gigabit symmetrical at the new site. Latency between the two locations isn't the most amazing ever at approximately 48 MS, but even still I'd think it would go a bit faster. I've tried both over the VPN tunnel and via the WAN externally just to test the speeds and performance is within 5% of each other so it's for sure not my VPN overhead. Any thoughts/suggestions on how to make this go faster as I have 5 nodes to empty and over 80 VMs. This is going to take forever! haha...

 

[BobhWasatch]

More data would be helpful. Have you tested the connection with iperf? You may have fast connections but with 48 ms of delay there are a lot of things in-between the sites.

 

[_gabriel]

try migrate VM shutdown to see if better

 

[waltar]

That looks to a lot of individual migration downtimes.
Btw. you could split snapshot images into reasonable filesizes and rsync compressed parallel, at the end shutdown vm, split again and rsync just need few small files. Could be done by script easily again in parallel to your 80 vm's - so it's just an idea to you. Good luck.

 

[Solinus]

try migrate VM shutdown to see if better

Yeah, I'm doing it with everything in offline mode already, but good thought.

That looks to a lot of individual migration downtimes.
Btw. you could split snapshot images into reasonable filesizes and rsync compressed parallel, at the end shutdown vm, split again and rsync just need few small files. Could be done by script easily again in parallel to your 80 vm's - so it's just an idea to you. Good luck.

I have to admit, I'm not super well versed in how to do that with rsync. I'll look into it and see if that's an option for me. Thanks!

More data would be helpful. Have you tested the connection with iperf? You may have fast connections but with 48 ms of delay there are a lot of things in-between the sites.

Yeah, it's interesting because one site routes to Denver (I'm in the Salt Lake City area) and the other goes out to Los Angeles before coming back. Seems silly to route the traffic that way especially since the locations are within 9 miles of each other physically. Honestly...I might just take a temporary NAS over there like a synology or something and back everything up to it, then restore it locally. A bit of a pain, but might be the only "fast" way.