Live Restore functionality

[tjk]

I can't find anything about what Live Restore actually does, can someone link me to docs that explain what it does and how it works?

Does it run the VM from the backup server and allow you to migrate to the PVE node, like Veeam and other vendors do?

 

[fiona]

Hi,
the documentation is here https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_live_restore
Essentially, the VM is started as usual on the Proxmox VE host, but if it reads a block that's not yet restored, it will be restored immediately. The usual (sequential) restore is running in the background.

 

[spf_tech]

Hi,
is there some technical information about how those live feature work internally ?
I guess it's shares the same concepts as live migration for VMware VMs.
Rather fascinating, but I have not been able to find anything except this manpage.
Regards

 

[fiona]

Hi,

is there some technical information about how those live feature work internally ?

it uses a QEMU block-stream job: https://qemu.readthedocs.io/en/v9.0.2/interop/qemu-qmp-ref.html#qapidoc-626
as well as a custom alloc-track QEMU block driver implementation: https://git.proxmox.com/?p=pve-qemu...c;hb=74466103898a3df6d343b22660474c18b6e930ac

I guess it's shares the same concepts as live migration for VMware VMs.

You are right, it also uses the same mechanism Well, getting the source block device for the stream job is different of course.

 

[spf_tech]

Thank you very much !

I was asking because I am testing live-migration from our Broadware infrastructure, which is backed up by Commvault.

When a backup task starts, the disk streaming is killed right after the creation of the snapshot, with the following error :

restore-scsi1: Cancelling block job
restore-scsi1: Done.
An error occurred during live-restore: block job (stream) error: restore-scsi1: Input/output error (io-status: ok)

But if I launch a live-migration and perform some snapshot creation/destruction on source side, there is no problem.

So I will read a little bit, just for my understanding.

Very nice feature so far !

Regards

 

[fiona]

When a backup task starts, the disk streaming is killed right after the creation of the snapshot, with the following error :

restore-scsi1: Cancelling block job
restore-scsi1: Done.
An error occurred during live-restore: block job (stream) error: restore-scsi1: Input/output error (io-status: ok)

Hmm, you might need to wait until the streaming is finished before doing other operations with the disk. In Proxmox VE, the VM will be locked with the create lock until streaming is finished, so you cannot do a Proxmox VE-initiated backup at the same time.

But if I launch a live-migration and perform some snapshot creation/destruction on source side, there is no problem.

Note that the live-migration from VMWare is not the same as a usual live-migration between Proxmox VE nodes. For the latter, there is no block-stream job.

 

[spf_tech]

Sometimes words are quite confusing

I mean when I launch a migration from VMware using live import, there is no problem if I perform snapshot operations on VMware.
According to what I see in the event log, Commvault client seems to do the same things.

But most probably it's also doing some extra stuff which causes disk streaming to abort when a backup task is running.

So as a conclusion : leave disks alone while they are streamed !