Do you need to shutdown VMs to backup them with stop mode?

[IsThisThingOn]

Do you need to shutdown VMs to backup them with stop mode? Or is it needlessly loosing uptime by doing so?

Here is how I used to think it works:
I thought it does not matter if the VM is running or not. In both cases the VM will be shutdown, a complete backup taken, during that the VM will be offline, after that the VM starts up again.

Since I often do snapshots when the VM is shutdown, and do a backup afterwards, or the backups take place automated at night, I never noticed that this is not true.

Here is how I now think it works:
A running VM will gracefully shutdown, when starting a backup in stop mode.
Since the shutdown VM now does not have an data in RAM, we can now create a dirty bitmap and get a consistent "reference point". We can immediately start up the VM again, way before the backup is finished.

What I still don't get:
Is the dirty bitmap there to just to know the state of the shutdown VM, and it will only backup the state of the VM when it was shut down?
Or is the dirty bitmap there to keep track of the changes and create a backup of the up and running VM again? Does that include RAM?

I think the former has to be true, otherwise would there not be again a small inconsistency risk involved?

 

[Onslow]

Hi, @IsThisThingOn
For the questions about stop mode I don't know the exact answers.

For the dirty bitmaps: from the docs ( https://pbs.proxmox.com/docs/technical-overview.html#fixed-sized-chunks ) and from the observation I know that dirty bitmaps are used only for disk images (not RAM) of VMs which are running.

If the VM was rebooted after the previous backup, the dirty bitmap of this VM is cleared / invalidated and it's created from scratch and only the next (not the current) backups' processes will profit from the bitmap (up to a next shutdown / reboot).

 

[Impact]

Is there a specific reason you use stop mode?

 

[IsThisThingOn]

and from the observation I know that dirty bitmaps are used only for disk images (not RAM) of VMs which are running.

That is what I thought, but I am not sure if this is entirely correct.
In the logs you can see this, no matter if the VM was running or not, when using stop mode:
INFO: scsi0: dirty-bitmap status: created new
So there is some dirty-bitmap that gets created. How that is used? I have no idea and would like to know.

Is there a specific reason you use stop mode?

Good question. Maybe that is based on wrong assumptions of mine.

For example, I have a Vaultwarden VM or Nextcloud VM that both use DBs. I know that in theory it not should be a problem, but even according to the docs, there is a small inconsistency risk. Since I am not in a hurry, and backups only take around 2min, I am not willing to take that risk. Especially considering that apparently QEMU is involved in the snapshot mode. QEMU recently destroyed a lot of trust for me. There is the DB high IO pressure problem with the newest version, but there is also the less talked about error https://forum.proxmox.com/threads/guest-agent-not-running-on-windows-11.175806/#post-818994 that even affects older versions. So even though I am not using Windows for these VMs, having to rely on QEMU working correctly for backups seems like a bad idea to me.

 

[Onslow]

In the logs you can see this, no matter if the VM was running or not, when using stop mode:
INFO: scsi0: dirty-bitmap status: created new
So there is some dirty-bitmap that gets created. How that is used? I have no idea and would like to know.

Dirty bitmap is used in snapshot mode (I don't know for suspend mode).

 

[IsThisThingOn]

Dirty bitmap is used in snapshot mode (I don't know for suspend mode).

but apparently also in stop mode. As seen by my output logs from stop mode. Sure it might not get used or used in a differently way, but it seams like it at least gets created?

 

[Onslow]

Yes, seems it is created. So either it didn't exist just before this backup or it existed but the current backup mode (stop mode) invalidated it. I don't know.
I would check what events will take place during the next stop-mode backup.
Will the bitmap be used or "created new" again?...

 

[Puliper]

In stop mode, Proxmox gracefully shuts down the VM for backup. Dirty bitmaps track disk changes only, not RAM. Use stop mode for DB VMs to avoid inconsistencies.

 

[IsThisThingOn]

Dirty bitmaps track disk changes only, not RAM.

Ahh, so it is only there to know what it needs to backup, to get a backup from the shutdown state, without the changes that happened to the disk during the backup.

So maybe on ZFS it would theoretically also be possible to create a snapshot and do a backup of that snapshot, while the VM is already up and running again and doing changes to the disk. Just like we have I guess now with the bitmap.