qmrestore - Hanging all other VM's?

[Ryan0751]

I have a single node PVE machine running 5.4-13.

When trying to restore a VM using qmrestore (either via the command-line or the web-UI, does not matter), when restoring a VM at some point during the restoration, my other VM's go unresponsive.

The PVE box is still reachable, but all VM's are unresponsive. qm commands time out. I had to power cycle the box as reboot wouldn't even cause the system to restart.

Interestingly, if I shut down ALL VM's prior to running qmrestore, the restoration works, but of course, this is not ideal.

The machine is running nowhere near fully loaded (the system is a 12-core Xeon, 128GB of RAM, and I have just 7 VM's running all at low CPU and memory utilization at the moment).

 

[Chris]

Hi,
you probably use most of the disk and/or network io during the restore, therefore increasing the iodelay for the other machines.
Please try to rate limit the qmrestore (see option --bwlimit in man qmrestore).
You can monitor IO using tools such as atop or iotop.
Hope this helps solve the issue.

 

[Ryan0751]

Hmm, ok, interesting. I can try that out.

If that is the case, shouldn't the GUI invocation of qmrestore have some sort of reasonable rate limiting specified by default?

Also, the system doesn't actually ever regain health once in this state. Everything stops, INCLUDING qmrestore.

 

[Chris]

Hmm, ok, interesting. I can try that out.

If that is the case, shouldn't the GUI invocation of qmrestore have some sort of reasonable rate limiting specified by default?

Also, the system doesn't actually ever regain health once in this state. Everything stops, INCLUDING qmrestore.

Hmm, okay than the issue might be somewhere else. Can you post the output of the syslog from around the time when this happens?